Infrastructure · 10 min
Hosting an open-source model:
what you really need to size.
On-premise, managed infrastructure or cloud: a practical look at capacity, costs and responsibilities.
MODELINFERENCEAPPLICATION
Key takeaways.
- 01
A model available for download is not automatically free of charge, unrestricted for all uses or suitable for production.
- 02
Model size is not enough: context, concurrency, latency and availability determine the infrastructure.
- 03
Total cost includes integration, monitoring, backups, energy and operational staff time.
The model is only one layer of the service.
Making a model useful requires an inference engine, API, identity management, interface, connectors, logs, an update strategy and monitoring. The weights file is therefore not a finished product.
The licence and model card also need checking: permitted uses, restrictions, language, maximum context, known biases and recommended hardware vary considerably.
- Model weights and licence
- Inference engine and quantisation
- CPU, GPU, memory and storage
- API, portal and authentication
- Connectors and document indexes
- Monitoring, backup and updates
Three options, three sets of responsibilities.
The right choice depends as much on your operations team as on data sensitivity. An underused server can cost more than managed capacity; a cheap API can become unsuitable when it handles a critical workflow.
MODELINFERENCEAPPLICATION
Technical questions that change the quotation.
A test with one person does not predict behaviour with twenty concurrent users. Measure the actual scenario, then retain headroom for peaks and maintenance.
- 01
How many active users and concurrent requests?
✓ - 02
What is the average query, context and answer size?
✓ - 03
What time to first response is acceptable?
✓ - 04
Does the system process text, images, audio or several modalities?
✓ - 05
What availability level is actually necessary?
✓ - 06
Can the model be reduced or quantised without losing useful quality?
✓ - 07
Do development, acceptance testing and production need separation?
✓ - 08
What volume growth is expected over six and twelve months?
✓
Calculate the full cost, not just the GPU price.
Add hardware or rental, energy, networking, storage, backups, maintenance, any licences and staff time. Then divide by the volume actually processed, using a conservative utilisation assumption.
For a first project, a managed environment often allows the use case to be validated before investing. On-site deployment becomes relevant when constraints, volume or infrastructure strategy justify it.
DECISION POINTA useful decision: size a service for a measured scenario, then plan how it will scale, rather than buying a machine around a model name.
Check and explore further.
We prioritise official texts and reference frameworks. This guide's recommendations are our practical interpretation of those sources, to adapt to your context.
- 01Hugging Face — Model Cards ↗
Documentation of declared model uses, limitations, licences and characteristics.
- 02ANSSI — Security of generative AI systems ↗
Security principles from development to production.
- 03NIST — Generative AI risk profile ↗
Risks and controls specific to generative systems.