Note: This retrospective is specifically about GPT-3.5, not other OpenAI models.
This is a retrospective on what it was like to use GPT-3.5 during its peak, why the experience could be so frustrating, and how it shaped my view of AI, tooling, and infrastructure.
How It Started
When I first discovered GPT-3.5, it felt revolutionary.
An AI that could write, explain code, answer questions, and help troubleshoot technical problems sounded incredible. I expected something that could assist with real-world projects and save me time.
Instead, I found myself dealing with one of the roughest periods of modern consumer AI.
The biggest problem wasn’t that GPT-3.5 made mistakes.
It was that it could make those mistakes with confidence.
Confidently Wrong Information
GPT-3.5 could produce answers that sounded authoritative while being completely incorrect.
I encountered situations where it:
- Incorrectly claimed that MyOwnFreeHost did not have an API, despite the API having existed since 2021.
- Gave inaccurate explanations about when that API was introduced.
- Claimed that Blooket did not have a game mode called Crypto Hack, despite the mode existing before the model’s stated knowledge cutoff.
- Correctly identified other Blooket game modes while incorrectly denying the existence of Crypto Hack.
- Generated URLs that did not exist or led to 404 pages.
- Invented product features, services, and historical details.
The frustrating part was the confidence.
Instead of simply saying “I don’t know,” the model could produce a detailed explanation for something that wasn’t true.
Phrases such as:
“As of my last knowledge update…”
or:
“As an AI language model…”
didn’t necessarily make the answer safer. In many cases, they simply surrounded an incorrect answer with language that made it sound more authoritative.
That distinction became important to me: sounding cautious isn’t the same thing as actually being reliable.
Made-Up Services and Categories
Some hallucinations were especially strange.
I encountered recommendations involving services that either didn’t exist, didn’t perform the function described, or were presented in contradictory ways.
For example, GPT-3.5 could recommend Freenom as a solution to a problem where Freenom wasn’t actually the appropriate tool, or even describe supposed alternatives that didn’t exist.
It could also describe paid services as though they were free.
These weren’t sophisticated reasoning failures. They were basic factual errors presented as recommendations.
And when you’re following technical advice, that matters.
A wrong explanation is annoying.
A wrong explanation that sends you to a nonexistent service or broken URL wastes even more time.
The Windows 3.1 Incident
One of the clearest examples of GPT-3.5 losing track of the actual problem happened when I asked about modern web browsing limitations on Windows 3.1.
Instead of explaining the technical constraints, GPT-3.5 started producing so-called “alternative solutions” that had little or nothing to do with the problem.
Among its suggestions was the idea of running Windows 3.1 inside a modern web browser as a workaround.
That doesn’t solve the underlying problem.
If the environment you’re emulating lacks the necessary modern networking, cryptography, browser-engine, and web-platform capabilities, putting that environment inside another browser doesn’t magically give it those capabilities.
The model also produced bizarre suggestions involving things such as physical clocks and biometric hardware.
Those concepts had no meaningful connection to the problem.
The actual limitations were much more straightforward:
- Windows 3.1 predates modern browser engines.
- Its networking environment was extremely limited by modern standards.
- Modern HTTPS/TLS requirements are far beyond what its original software ecosystem was designed to support.
- Modern HTML, CSS, and JavaScript engines require vastly more capable software and hardware environments.
- Even if networking were partially solved, the lack of a contemporary browser engine would remain a fundamental problem.
The important lesson wasn’t simply that GPT-3.5 got one technical question wrong.
It demonstrated a broader failure mode:
Take several concepts that sound technical, combine them into a plausible-looking answer, and hope the result resembles a solution.
The output could look sophisticated while failing to address the actual engineering problem.
Mixing Up Layers of the Technology Stack
GPT-3.5 also had a tendency to mix up completely different layers of technology.
For example, it could confuse:
- DNS with HTTP routing
- Domain registration with hosting
- URL paths with subdomains
- Browser components with operating-system components
- WebView with an actual browser application
I ran into this while working on a subdomain service of my own.
I needed precise explanations of how domains, DNS, routing, and HTTP actually interacted.
Instead, I sometimes received architectural explanations that sounded reasonable until I examined what they were actually saying.
A path such as:
example.com/user
doesn’t become a DNS subdomain simply because /user identifies a particular user.
DNS operates on domain names.
HTTP paths are a different layer.
That distinction is basic infrastructure knowledge, but GPT-3.5 could blur it.
Fire OS and WebView
Another example involved Android-derived systems such as Fire OS.
GPT-3.5 could make incorrect claims about Android System WebView and suggest that Amazon’s Silk Browser effectively replaced WebView.
That’s an important distinction.
A browser application and the system components used by applications to render web content aren’t necessarily interchangeable.
Again, the problem wasn’t merely one wrong fact.
The model often seemed to understand the general vocabulary while failing to preserve the underlying architecture.
Broken Code That Couldn’t Be Fixed
Code generation was one of the biggest sources of frustration.
GPT-3.5 could produce code that looked convincing at first glance but contained:
- Incorrect APIs
- Outdated libraries
- Missing imports
- Invalid function names
- Incorrect assumptions about framework behavior
- Logic errors
- Incompatible versions
The most frustrating part was what happened afterward.
I’d point out the error and ask it to fix the code.
Sometimes it would simply repeat the same mistake.
Other times it would invent a new function or API.
Occasionally it would confidently say that the revised version “should work” when the underlying problem was still there.
That created a dangerous feedback loop:
- Generate code.
- Code doesn’t work.
- Ask the model to fix it.
- Model changes something unrelated.
- New code breaks somewhere else.
- Repeat.
For experimentation, this could be amusing.
For an actual project, it was exhausting.
Hardware and Device Chaos
The problem wasn’t limited to software.
At one point, GPT-3.5 gave me a Konami-style remote-control sequence that was supposedly intended to enable developer mode on a television.
Instead of enabling the desired feature, the sequence rebooted the device.
This was one of the moments when I became much more skeptical about following AI-generated instructions on physical hardware.
A model doesn’t inherently understand the consequences of a sequence just because it can describe the sequence fluently.
When instructions involve:
- Firmware
- BIOS settings
- Developer modes
- Factory resets
- Network configuration
- Physical devices
being “probably right” isn’t good enough.
Trying to Get It Fixed
For months, I tried to get the problems noticed.
I used:
- Dislike buttons
- Support messages
- Developer forums
- Reddit discussions
- Trustpilot reviews
- App-store reviews
The goal wasn’t simply to complain.
I wanted the problems to reach someone who could actually improve the system.
But a model doesn’t change because one person reports an incorrect answer.
There are evaluation pipelines, model versions, product priorities, safety systems, infrastructure constraints, and countless other factors involved.
From my perspective, though, the experience was much simpler:
I had a tool that repeatedly wasted my time, and I had very little control over fixing it.
But Wait, Can You Just Build Your Own LLM?
Technically, yes.
You can train a small language model yourself. You can experiment with hundreds of millions of parameters, and with sufficient hardware and expertise, even models around the billion-parameter range are possible outside the largest companies.
But that isn’t equivalent to recreating GPT-3.5.
A small personal model doesn’t automatically have:
- Broad world knowledge
- Strong coding ability
- Large-scale pretraining
- Extensive instruction tuning
- Robust evaluation
- The same inference infrastructure
- The same general-purpose capabilities
A personal model can be an excellent learning project.
It just isn’t a drop-in replacement for a large general-purpose assistant.
That distinction is important.
Can You Fine-Tune GPT-3.5?
Fine-tuning can be useful when you’re building a specific application.
You can customize a model’s behavior for a particular domain, style, or task.
But fine-tuning your own model doesn’t modify the public ChatGPT model.
It doesn’t mean that you’ve fixed GPT-3.5 for everyone.
And it doesn’t magically eliminate the underlying model’s tendency to produce incorrect information.
Fine-tuning is customization, not a universal repair mechanism.
Can Mixture-of-Experts Make It Cheap?
Mixture-of-Experts, or MoE, is an important technique for making large models more computationally efficient.
Instead of activating every parameter for every token, an MoE architecture can route each token through only a subset of experts.
That can reduce the amount of computation required per token.
But it doesn’t mean the inactive parameters don’t exist.
The complete model still has to be stored, trained, distributed, and served.
That creates a distinction between:
Total parameters
and
parameters activated for each token.
MoE can make computation more efficient, but it doesn’t magically eliminate memory, storage, training, or infrastructure costs.
What About Quantization and Offloading?
Quantization can reduce model memory requirements by representing weights with fewer bits.
Offloading can move parts of a model between GPU memory, system RAM, or other storage.
Both techniques are useful.
Neither is magic.
Quantization introduces accuracy and engineering tradeoffs, while aggressive offloading can significantly reduce performance because data has to move between memory tiers.
These methods can make inference possible on hardware that otherwise couldn’t hold a model.
They don’t turn a frontier-scale training project into something a typical individual can casually run on a home computer.
Can You Just Use Common Crawl?
This was another lesson I learned the hard way.
Common Crawl is free to access.
That does not mean building a training dataset from it is free.
The raw crawl data can be downloaded from public infrastructure, but actually turning it into useful training data requires:
- Storage
- Bandwidth
- Compute
- Extraction
- Filtering
- Deduplication
- Language identification
- Quality filtering
- Tokenization
- Dataset generation
And every processing stage can create additional intermediate data.
The raw dataset being publicly available doesn’t remove the infrastructure cost of processing it.
The “Just Build Your Own LLM” Myth
One of the most common responses I encountered was:
“Just build your own model.”
In theory, that’s empowering advice.
In practice, it can be wildly disconnected from the resources required.
There is a huge difference between:
training a small language model as a learning exercise
and:
building a competitive general-purpose assistant.
The first can be done by individuals.
The second requires enormous amounts of data, compute, engineering, evaluation, and operational infrastructure.
The exact cost depends heavily on the model, dataset, hardware, training strategy, number of experiments, and infrastructure.
So saying “just train your own LLM” without specifying the target is almost meaningless.
If someone means a small educational model, sure.
If they mean “build your own replacement for GPT-3.5,” that’s an entirely different project.
Trying to Build My Own AI
Eventually, I tried anyway.
That was when the infrastructure problem became impossible to ignore.
I experimented with ideas involving:
- Common Crawl
- Cloud storage
- Cloud compute
- IPFS
- Mixture-of-Experts
- Smaller open-source models
- Custom datasets
- Training frameworks
The costs added up quickly.
At one point, my estimates for storage and processed Common Crawl data were around $400 per month, depending on the workflow and how much derived data I retained.
And storage was only one part of the problem.
There was also:
- Compute
- Temporary storage
- Bandwidth
- Preprocessing
- Deduplication
- Indexing
- Training
- Experimentation
The important realization was that infrastructure costs aren’t a single line item.
They compound.
A dataset can be free.
Processing it isn’t necessarily free.
A model can be open source.
Training it isn’t necessarily free.
A cloud machine can be rented by the hour.
Running it continuously is still expensive.
Grants Don’t Magically Solve It
People also suggested grants.
Grants absolutely exist, but they’re generally aimed at researchers, institutions, universities, labs, nonprofits, and other organized projects.
They aren’t a universal mechanism that lets an individual with a personal computer obtain the infrastructure required to train a frontier-scale model.
That distinction matters.
“Funding exists” and “I can realistically obtain and sustain the funding necessary for this project” are two very different statements.
The Common Crawl Reality Check
I eventually contacted Common Crawl directly.
The basic suggestion was to work with a single crawl.
That may be perfectly reasonable for experimentation, but even one large crawl can represent a substantial data-processing project.
This was another point where online discussions sometimes missed what I was actually talking about.
People would respond to questions about cost and infrastructure with theoretical statements about what was technically possible.
Technically possible and economically practical are not the same thing.
The Myth of Common Crawl Alternatives
After running into the cost problem, I started looking for alternatives.
Surely there had to be another massive public dataset containing enough web-scale text to replace Common Crawl.
That turned out to be much more complicated.
A lot of things described online as “Common Crawl alternatives” aren’t actually equivalent.
They might be:
- Search APIs
- News datasets
- Curated feeds
- Academic corpora
- Web archives
- Wikipedia dumps
- Older research datasets
Those resources can be extremely useful.
But they don’t necessarily provide the same combination of scale, breadth, recency, and accessibility as a continuously collected web crawl.
That’s an important distinction.
An archive can be useful without being a replacement.
A search engine can expose information without giving you a raw training corpus.
Wikipedia can provide high-quality text without representing the entire public web.
It Wasn’t Just AWS
Another misconception is that the problem was specifically AWS pricing.
It wasn’t.
AWS can host public datasets through programs such as its Open Data initiative, meaning users don’t necessarily pay the normal storage cost for the provider’s hosted copy of the public dataset.
But that doesn’t make an entire data-processing pipeline free.
Once you start:
- Extracting text
- Filtering documents
- Deduplicating data
- Creating training shards
- Storing intermediate results
- Generating indexes
- Running preprocessing jobs
you create your own computational and storage requirements.
Those costs can exist on AWS, Google Cloud, Azure, or other infrastructure providers.
The exact price depends heavily on the architecture and retention policy.
In my own estimates, storage and derived data could reach hundreds of dollars per month and potentially much more.
And that was before adding substantial compute and training costs.
So the real problem wasn’t:
“AWS is expensive.”
It was:
Large-scale data processing is expensive.
You Can’t Simply Ignore the Data Problem
Another popular suggestion was:
“You don’t need that much data.”
For experimentation, that’s true.
For building a broadly capable language model, the situation is different.
A smaller dataset can produce a useful model, but reducing the dataset also changes what the model can learn.
There is no simple equation where:
tiny dataset + clever architecture = frontier model.
Training quality depends on much more than parameter count.
It involves:
- Data quality
- Data diversity
- Dataset size
- Model architecture
- Optimization
- Compute
- Training duration
- Instruction tuning
- Evaluation
- Post-training
A small model can be genuinely useful.
It just shouldn’t be confused with a general-purpose frontier model.
Experimenting With nanoGPT
After realizing that training a large model wasn’t realistic for me, I experimented with smaller frameworks such as nanoGPT.
The appeal was obvious.
It was relatively simple, understandable, and useful for learning how transformer language models actually work.
But that simplicity also highlighted the limitations.
nanoGPT is excellent for:
- Learning
- Experiments
- Educational projects
- Small-scale demonstrations
- Understanding transformer training
It isn’t intended to magically turn a personal computer into a frontier-model training cluster.
I could optimize the training process.
I could modify the architecture.
I could experiment with datasets.
But optimization couldn’t eliminate the underlying requirements for data and compute.
That was a humbling realization:
Better engineering can improve efficiency. It cannot repeal the laws of scale.
The Myth of “Good ChatGPT Alternatives”
I also spent a lot of time looking for alternatives.
At the time, many products marketed themselves as ChatGPT alternatives.
But they varied enormously.
Some were:
- Interfaces over GPT models
- Search-oriented systems
- Applications built on weaker models
- Specialized assistants
- Services with strict quotas
- Products with limited context windows
- Experimental systems
Some were genuinely useful.
But finding a single service that simultaneously provided broad knowledge, strong coding ability, conversational fluency, reliability, affordability, and openness was difficult.
This taught me another lesson:
An AI product is not just its interface.
Two applications can look almost identical while having completely different models and infrastructure underneath.
The chat box isn’t the intelligence.
The model, training, retrieval systems, inference stack, evaluation, and product engineering are what determine the experience.
The Turning Point
Eventually, the ecosystem started changing.
GPT-4 improved the situation substantially.
Then came newer models and a much broader ecosystem of tools and open-weight systems.
Technologies such as:
- GPT-4 Turbo
- GPT-4o
- OpenRouter
- Open-weight models
- Ollama
- Local inference
- Better retrieval systems
made experimentation much more accessible.
The important change wasn’t that hallucinations suddenly disappeared.
They didn’t.
The difference was that the overall ecosystem became much better at giving users access to stronger models, alternative backends, retrieval, local inference, and more flexible tooling.
For me, AI finally started feeling less like a single locked service and more like an actual technology ecosystem.
How Large Was GPT-3.5?
Another misconception I encountered was the claim that GPT-3.5 contained something like “100,000+ quadrillion parameters.”
No.
The original GPT-3 model was widely reported as having 175 billion parameters.
GPT-3.5 was a later model family or iteration rather than a model publicly documented as having an astronomically larger parameter count.
The exact architecture and parameter count of the various GPT-3.5 models were not publicly documented in the same way as GPT-3.
But the important point remains:
GPT-3.5 was not secretly an incomprehensibly enormous model.
And parameter count alone doesn’t determine reliability.
A model can be extremely large and still produce incorrect information.
Scale provides capacity.
It doesn’t provide a guarantee of truth.
“Just Train It on Something Specific”
Another suggestion I heard repeatedly was:
“Why don’t you just build a small AI and train it on something specific?”
That advice isn’t inherently wrong.
A specialized model can be extremely useful.
But it solves a different problem.
If the goal is a model that answers questions across programming, operating systems, hardware, networking, history, science, and everyday topics, training a tiny model on one narrow dataset isn’t going to reproduce that breadth.
A specialized model may perform well inside its intended domain while becoming unreliable outside it.
That’s not a failure.
That’s simply specialization.
The problem was that I wasn’t looking for a specialized classifier or domain chatbot.
I wanted a general-purpose assistant.
For that use case, the relevant requirements include:
- Broad pretraining
- Diverse data
- Strong language modeling
- Instruction tuning
- Post-training
- Evaluation
- Retrieval when appropriate
- Mechanisms for reducing unsupported claims
A small domain-specific model isn’t a substitute for all of that.
What I Actually Needed
Looking back, I can describe the problem much more clearly.
I didn’t need to build another GPT-3.5.
I needed a system that combined:
A capable base model
with
grounding and retrieval when current or specialized information was required
and
mechanisms that made the model less likely to confidently invent information.
That distinction changed how I thought about AI.
The solution to hallucination isn’t necessarily “train a smaller model.”
Sometimes the solution is better retrieval.
Sometimes it’s better evaluation.
Sometimes it’s a stronger model.
Sometimes it’s simply teaching the system to say:
“I don’t know.”
Looking Back
GPT-3.5 didn’t just frustrate me.
It taught me a lot.
It taught me:
- How unreliable early AI could be
- Why fluent language isn’t the same as factual accuracy
- How easily technical terminology can hide a broken explanation
- How expensive large-scale infrastructure really is
- Why open models and local inference matter
- Why retrieval and grounding are important
- Why model size alone doesn’t determine quality
- Why experimentation and production are completely different problems
- Why timing matters in technology
Ironically, the frustration also pushed me deeper into technology.
It contributed to me starting a tech YouTube channel and learning much more about AI, infrastructure, operating systems, networking, and software engineering than I probably would have otherwise.
Conclusion
I don’t regret learning about AI.
I regret discovering it during a period when the technology was still rough enough that the gap between sounding intelligent and being reliable was enormous.
GPT-3.5 was capable enough to make you believe it could solve almost anything.
It wasn’t reliable enough to actually do that.
That combination was uniquely frustrating.
The problem wasn’t that GPT-3.5 was useless.
It was that it was useful enough to trust and unreliable enough to punish that trust.
Over time, the ecosystem changed.
Models improved. Open-weight systems became more accessible. Local runtimes appeared. Retrieval became more common. Tool use became more sophisticated. Developers gained more control over which models and infrastructure they used.
The problems I spent months fighting didn’t disappear because I somehow found the perfect workaround.
They became less important because the technology moved forward.
And that’s probably the biggest lesson I took from the entire experience:
Sometimes the solution to a technology problem isn’t finding a better workaround.
Sometimes you just have to wait for the technology to mature.
