Instructions to use BucketOfFish/simplified_phi2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use BucketOfFish/simplified_phi2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="BucketOfFish/simplified_phi2", trust_remote_code=True)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("BucketOfFish/simplified_phi2", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use BucketOfFish/simplified_phi2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "BucketOfFish/simplified_phi2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BucketOfFish/simplified_phi2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/BucketOfFish/simplified_phi2

SGLang

How to use BucketOfFish/simplified_phi2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "BucketOfFish/simplified_phi2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BucketOfFish/simplified_phi2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "BucketOfFish/simplified_phi2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BucketOfFish/simplified_phi2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use BucketOfFish/simplified_phi2 with Docker Model Runner:
```
docker model run hf.co/BucketOfFish/simplified_phi2
```

BucketOfFish commited on Jan 2, 2024

Commit

7da2fc9

1 Parent(s): dc6124b

Removed output check

Browse files

Files changed (1) hide show

streaming_inference.py +0 -17

streaming_inference.py CHANGED Viewed

@@ -70,20 +70,3 @@ if __name__ == "__main__":
         my_output += new_token
         print(new_token, end="", flush=True)
     print()
-    # check output
-    expected_output = """Here is an essay on sea monkeys:
-Sea monkeys are a type of brine shrimp that are often sold as pets in kits. They are easy to care for and can be a fun addition to any aquarium. However, it is important to understand the proper care and feeding of sea monkeys to ensure their health and longevity.
-Sea monkeys are sensitive to changes in temperature, pH, and salinity. It is important to keep their environment at a consistent temperature of around 75-80 degrees Fahrenheit. The pH level should be between 7.5 and 8.5, and the salinity should be around 1.023. These conditions can be achieved by using a saltwater mix and a thermometer.
-Feeding sea monkeys is also important. They can be fed a variety of foods, including flakes, pellets, and freeze-dried bloodworms. It is important to feed them regularly, but not too much. Overfeeding can lead to health problems and a shorter lifespan.
-In addition to proper care and feeding, it is important to understand the different types of sea monkeys. There are several different species, including the brine shrimp, the fairy shrimp, and the tadpole shrimp. Each species has its own unique characteristics and requirements.
-Sea monkeys are also known for their ability to reproduce quickly. They can lay hundreds of eggs at a time, which can lead to a large population in a short amount of time. However, it is important to keep their environment clean and free of debris to prevent overcrowding and disease.
-In conclusion, sea monkeys are a fascinating and easy-to-care-for pet. However, it is important to understand their unique requirements and to provide them with proper care and feeding. By doing so, you can ensure their health and longevity and enjoy watching them grow and thrive in their aquarium.<|endoftext|>"""
-    assert my_output == expected_output
-    print("Test passed")

         my_output += new_token
         print(new_token, end="", flush=True)
     print()