OSError: You seem to have cloned a repository without having git-lfs installed
Asked by Jabalnur:

I have pushed a FastAPI program that have large file. The repository is successfully pushed, but when the program was running, there is an error about git-lfs.   

<br />

Traceback (most recent call last):  
15  
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/transformers/modeling_utils.py", line 460, in load_state_dict  
16  
return torch.load(checkpoint_file, map_location="cpu")  
17  
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/torch/serialization.py", line 815, in load  
18  
return _legacy_load(opened_file, map_location, pickle_module, \*\*pickle_load_args)  
19  
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/torch/serialization.py", line 1033, in _legacy_load  
20  
magic_number = pickle_module.load(f, \*\*pickle_load_args)  
21  
_pickle.UnpicklingError: invalid load key, 'v'.  
22  
23  
During handling of the above exception, another exception occurred:  
24  
25  
Traceback (most recent call last):  
26  
File "/root/.local/share/virtualenvs/build-3vGKWv3F/bin/uvicorn", line 8, in \<module\>  
27  
sys.exit(main())  
28  
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/click/core.py", line 1157, in __call__  
29  
return self.main(\*args, \*\*kwargs)  
30  
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/click/core.py", line 1078, in main  
31  
rv = self.invoke(ctx)  
32  
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/click/core.py", line 1434, in invoke  
33  
return ctx.invoke(self.callback, \*\*ctx.params)  
34  
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/click/core.py", line 783, in invoke  
35  
return __callback(\*args, \*\*kwargs)  
36  
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/uvicorn/main.py", line 416, in main  
37  
run(  
38  
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/uvicorn/main.py", line 587, in run  
39  
server.run()  
40  
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/uvicorn/server.py", line 61, in run  
41  
return asyncio.run(self.serve(sockets=sockets))  
42  
File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run  
43  
return loop.run_until_complete(main)  
44  
File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete  
45  
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/uvicorn/server.py", line 68, in serve  
46  
config.load()  
47  
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/uvicorn/config.py", line 467, in load  
48  
self.loaded_app = import_from_string(self.app)  
49  
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/uvicorn/importer.py", line 21, in import_from_string  
50  
module = importlib.import_module(module_str)  
51  
File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module  
52  
return _bootstrap._gcd_import(name\[level:\], package, level)  
53  
File "\<frozen importlib._bootstrap\>", line 1050, in _gcd_import  
54  
File "\<frozen importlib._bootstrap\>", line 1027, in _find_and_load  
55  
File "\<frozen importlib._bootstrap\>", line 1006, in _find_and_load_unlocked  
56  
File "\<frozen importlib._bootstrap\>", line 688, in _load_unlocked  
57  
File "\<frozen importlib._bootstrap_external\>", line 883, in exec_module  
58  
File "\<frozen importlib._bootstrap\>", line 241, in _call_with_frames_removed  
59  
File "/opt/atlassian/pipelines/agent/build/naturalness_score/api.py", line 6, in \<module\>  
60  
from .classifier.model import Model, get_model  
61  
File "/opt/atlassian/pipelines/agent/build/naturalness_score/classifier/model.py", line 110, in \<module\>  
62  
model = Model()  
63  
File "/opt/atlassian/pipelines/agent/build/naturalness_score/classifier/model.py", line 23, in __init__  
64  
self.model = DistilBertForMaskedLM.from_pretrained(config\["PRE_TRAINED_MODEL"\],config=self.config)  
65  
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2629, in from_pretrained  
66  
state_dict = load_state_dict(resolved_archive_file)  
67  
File "/root/.local/share/virtualenvs/build-3vGKWv3F/lib/python3.10/site-packages/transformers/modeling_utils.py", line 465, in load_state_dict  
68  
raise OSError(  
69  
OSError: You seem to have cloned a repository without having git-lfs installed. Please install git-lfs and run \`git lfs install\` followed by \`git lfs pull\` in the folder you cloned.  
70


Answers:
author: Patrik S
body: Hello [@Jabalnur](/t5/user/viewprofilepage/user-id/5294368) and welcome to the Community!

Based on the logs you have shared, it seems like you are using Bitbucket Pipelines to build/run your code.

When a Pipeline starts, it will automatically clone the repository to a folder inside the build container, but by default, this clone does **not include LFS files**.

It seems like the program you are executing in pipelines also needs the files that are stored in the LFS portion of your repository. To instruct pipelines to also clone the LFS during its startup you can include the **lfs: true** attribute in the bitbucket-pipelines.yml file.

You have the option to specify this at the global level (applies to every step) or at the step level (applies only to the step where the attribute was included) :

* **Global level** : Using lfs to download all LFS files in every step :   

  ```
  clone:
    lfs: true
  pipelines:
    default:
      - step:
          name: Clone and download
          script:
            - echo "Clone and download my LFS files!"
  ```

* **Step level** : Using lfs to selectively download LFS files  

  ```
  pipelines:
    default:
      - step:
          name: Clone with lfs on
          clone:
            lfs: true
          script:
            - ls -lh large-file.zip # 26M large-file.zip
      - step:
          name: Clone with lfs off
          clone:
            lfs: false
          script:
            - apt-get update && apt-get install -y git-lfs
            # Download only desired files
            - git lfs pull --include=large-file.zip 
            - ls -lh large-file.zip # 26M large-file.zip
  ```

You can find more details on the usage of LFS attribute in the [Pipelines \| Git Clone Behaviour \| LFS documentation](https://support.atlassian.com/bitbucket-cloud/docs/git-clone-behavior/#LFS)

Let me know in case you have any questions.

Thank you, [@Jabalnur](/t5/user/viewprofilepage/user-id/5294368) !

Patrik S

comments: author: Jabalnur
body: Here is my code, could you check it base on your suggestion before. Please let me know if there is anything wrong or another thing I can add.  

image: python:3.10

pipelines:  
default:  
- parallel:  
- step:  
name: 'Build and Test'  
script:  
- echo "Your build and test goes here..."  
- step:  
name: 'Lint'  
script:  
- echo "Your linting goes here..."  
- step:  
name: 'Security scan'  
script:  
- echo "Your security scan goes here..."

- step:  
name: 'Deployment to Staging'  
deployment: staging  
script:  
- echo "Your deployment to staging script goes here..."  
# Install pipenv  
- pip install pipenv  
# Remove any existing Pipfile.lock  
- rm Pipfile.lock  
# Install dependencies from Pipfile  
- pipenv install --deploy

# Start UVicorn server  
- echo "Starting UVicorn server..."  
- pipenv run uvicorn naturalness_score.api:app --host=0.0.0.0 --port=8000

- step:  
name: 'Deployment to Production'  
deployment: production  
trigger: 'manual'  
script:  
- echo "Your deployment to production script goes here..."


author: Jabalnur
body: Finally I succeeded in deploying my API to deployment, but how do I use the endpoints?

![Screenshot 2023-08-28 at 23.50.36.jpg](https://community.atlassian.com/t5/image/serverpage/image-id/277591iFE8F8AC6268E1354/image-size/large?v=v2&px=999 "Screenshot 2023-08-28 at 23.50.36.jpg")


author: Jabalnur
body: my deployment to production is still loading until now, Do you have any solution to this problem? and how to call the API which I use FastAPI? This is my API code:  

# from typing import Dict

from fastapi import Depends, FastAPI  
from pydantic import BaseModel

from .classifier.model import Model, get_model

app = FastAPI()

<br />

class NaturalnessRequest(BaseModel):  
text: str

class Config:  
orm_mode = True

<br />

class NaturalnessMaskResponse(BaseModel):  
text_mask_fill: str

<br />

class NaturalnessPerplexityResponse(BaseModel):  
perplexity: float

<br />

class NaturalnessRetrainResponse(BaseModel):  
status: str

<br />

<br />

[@app](/t5/user/viewprofilepage/user-id/1607004).post("/fill_mask", response_model=NaturalnessMaskResponse)  
async def fill_mask(request: NaturalnessRequest, model: Model =  
Depends(get_model)):  
text_mask_fill = model.fill_mask(request.text)  
return NaturalnessMaskResponse(  
text_mask_fill=text_mask_fill  
)

<br />

[@app](/t5/user/viewprofilepage/user-id/1607004).post("/perplexity", response_model=NaturalnessPerplexityResponse)  
async def perplexity(request: NaturalnessRequest, model: Model = Depends(get_model)):  
perplexity = model.perplexity(request.text)  
return NaturalnessPerplexityResponse(  
perplexity=perplexity  
)

[@app](/t5/user/viewprofilepage/user-id/1607004).post("/retrain", response_model=NaturalnessRetrainResponse)  
async def retrain(request: NaturalnessRequest, model: Model = Depends(get_model)):  
status = model.retrain(request.text)  
return NaturalnessRetrainResponse(  
status=status  
)


Tags: bitbucket-cloud

cloud