This November, I completed my internship at Vivasoft Nepal as an Associate AI/ML Engineer. This blog post is a retrospective of my experience and (almost) everything I learnt during my internship.
Part I : Getting Into the company
As a final year university student majoring in Computer Science, I was very obviously stressed about getting an internship in a good company. My process of applying to companies was an unorganized chaos. I scrolled LinkedIn and any other job portals all day and applying to any and all internship roles I could find.
As far as I can recall, I must’ve applied to 35 companies before stumbling onto the job opening for Vivasoft. I immediately applied for the position and all I could do was wait.
The AI Interview
Some days later, I received an email for a “Pre-Screen AI Interview”. My immediate reaction to the email was “This company is a scam. They are using this job offering to just collect data for their internal system“. To be fair, LinkedIn was filled with these fake job openings from companies like micro1. I had made up my mind not to join the interview. Until one day, when I was bored out of my mind, I decided to get the interview over with. Now remember, I still did not think the job opening was legit. I got out of my bed, did not change clothes and did not make myself pleasant to look at in any way. I just clicked the link and started the interview.
The interview was around 30 minutes long and to my surprise, the questions were pretty challenging and kind of difficult to answer. The questions mostly revolved around the projects I had already done. The system was fast and polished. I answered all the questions to the best of my knowledge and was feeling pretty good. Maybee…. just maybe, the job opening was legit?
The Online Interview
Around a week later, I was chilling with my brother and having some sekuwa with him. I received a call from an unknown number. The person on the other side was Pushpa Dai from Vivasoft Nepal. He informed me that I had passed the pre-screening process and next in line was an online interview with the Bangladesh team.
“HOLY SH\T, THE JOB IS REAL. THIS IS REAL“. Now remember, I had applied to 35 companies beforehand, only heard back from a few of them and was rejected *EVERY. SINGLE. TIME. I was understandably nervous. The worst part is, the interview was scheduled only 15 mins after my Deep Learning elective quiz. No worries, I have done this before, I could do this once again.
I was interviewed by Aminul Bhai, Munna Bhai and Rafael Bhai. The interview was around 30 minutes long and pretty technical. Most technical questions were asked by Aminul bhai. The biggest issue with employing me would be the fact that I was still in university, and my university is at least 2 hours away from any office in Kathmandu. If I were to work, I would have to work remotely for a huge part of my time. I was rejected before because of this issue as well. I had a gut feeling that I had aced the interview, but again my college and timing would be a liability.
I was expecting a rejection mail a couple of days later. But, cut to 6 hours later, I got an offer letter.
I am not exaggerating when I say I was trembling from shock. I finally did it. After months and months of applying to everything I come across, I did it.
Like the email said, I went to the office, had dinner with one of the co-founders, Tareq Bhai. I also met everyone in the team the same day. The office was nice, the people were welcoming, and remote work was flexible. I immediately felt at home with the company and the warmth with which people welcomed all the new interns. Everyone was going to begin working ASAP, but since I had my board exams, I would be joining from the first of August.
What’s going to stay with me
I did learn some things just from the interview and on boarding process.
Never say no
If you remember, I was initially not going to “waste my time“ with the AI interview. The only reason I got into the company is because I said to myself “F\ck it, let’s just do it“*.
Now that I think of it, most things I have achieved in my life till now are just a result of me being one of the people who didn’t want to say no. Take your chances, do not be afraid of failing.
Build what you like
Kathmandu University requires students to build projects each semester for 5 semesters. Most people build what they see other people building. People mostly end up building “XYZ Management Systems“ because these projects are the simplest to conceptualize and to explain to the evaluator.
My approach to building was always different. I’d first decide what new concepts for tech I wanted to learn. Then, I’d build an idea around these concepts. This meant that, I always had a deep understanding of what I was building and why everything was built that way. I always had a deep technical knowledge of the things of the processes I chose and the frameworks I used. I can’t say for sure, but I believe that was a major factor for Vivasoft Nepal accepting me into the company.
Part II : The Work
I joined the company from the first of August when everyone else had joined 2 weeks ago. Like most early days in any internship, I did not have much to do. The other guys were working on some STS pipeline R&D but I kind of did not have anything to do. All I did was just talk to people, drink coffee and have office lunch.
A few days after I joined, Shaf bhai, the CEO along with Talha Bhai and Munna Bhai came to the Nepal office. We had a small welcome party and some dinner.
LLM Finetuning
The next day, Talha bhai FINALLY assigned me a task. He wanted me to explore Unsloth and how to finetune an LLM on synthetic data. I had never worked with LLMs before, they felt just too complex and I felt like I never had enough compute to work with them.
Boy was I wrong. Unsloth makes everything so much easier and efficient. I learnt about concepts like PEFT and LoRA. Talha bhai had asked me fine tune Gemma 1b on a synthetic Nepali dataset and report to Sadman bhai. Sadman bhai also gave me access to the company Wandb account and told me about his earlier findings.
I used the Gemini API to create some synthetic conversational data about tourism in Nepal in the Nepali language.
Then I proceeded to fine tune Gemma 1b using Unsloth on Google colab. I did encounter some problems along the way, but nothing I couldn’t fix. The model began to learn quite nicely actually. Then, I did some manual testing and the outputs were better than I expected.
The model had really great Nepali grammar and sentence flow although it got some facts wrong. I experimented with some parameters and reported my findings to the Dhaka team.
LLM Inference Server
Fine tuning an LLM is useless if you can’t do cool stuff with it. Sadman bhai already had the skeleton of an inference server ready, but it was no where close to being production ready. I was given the responsibility of the inference server next.
OpenAI Compatibility & LoRA Comparison
The OpenAI API specification is the industry standard for LLM inference servers. My initial job was to make the current server OpenAI compatible. In doing this, I understood the specification and the different functions an LLM server must be capable of very nicely.
But, apart from the basic OpenAI specs, Talha bhai wanted a “Bring Your Own Adapter“ feature on the server. Basically, the user must be able to specify what LoRA adapter they want to attach to the LLM for the response. He also wanted a simple comparison UI where one could compare the outputs from any two adapters for a single prompt.
I got everything done pretty quickly, but….
The current server was not fit to handle these features. Loading and unloading LoRA adapters is very expensive computationally. Also, the current server was unable to handle any API calls in parallel, so the inference time for any query was higher than 1000 seconds!
Production Ready LLM Inference Server
vLLM is a very amazing tool that makes creating production ready and OpenAI Compatible LLM servers as easy as running a single terminal command. It is a batteries-included framework with streaming, parallel inference, GPU utilization control and many features pre-packaged.
Since Talha bhai had very specific requirements regarding the adapter swapping, I couldn’t simply use the vLLM CLI. I had to write some code myself to ensure vLLM fits into our pre-existing server. Sadly, vLLM requires a CUDA enabled device and my laptop only has a CPU. A remote GPU was arranged and I began writing code on a remote server.
This was my first ever experience working on something like this and I had lots of fun working with this beefy GPU. I may or may not have used the GPU time to tinker around with some other models as well.
In the end, after integrating vLLM into the server, the inference time improved drastically and the server was production ready.
Let’s just say I am most proud of this task I completed during my internship period.
Previa (Gaze Detection)
Remember that AI interview portal I spoke about earlier, yeah that is Previa. The system is extremely feature rich but I found a gap and wanted to contribute to the project. The system lacked a way to reliably determine whether a interviewee was cheating or looking at their phone or not.
So, I asked Munna bhai how I could contribute this feature to the system. He told me to get a small demo and document ready, and I did exactly that.
I quickly created a small demo that tracked the interviewee’s face using pre-existing MediaPipe models.
For how little effort it required, the system actually performed better than I expected. Best part is, all the DL stuff happens on the client side (60 FPS btw) and not many changes are required on the backend.
VivaDai
For a huge chunk (hah, RAG reference) of time, we 4 AI Interns were working on small isolated tasks. Until one day, Talha bhai and Munna bhai asked us to work together on a PoC project. They explained us their vision for an “organizational memory management system“. I probably won’t touch on the intricacies of the project that much in this section.
The AI team in Nepal spent days and days playing with the concept and designing a system. Since we always have that one ‘dai‘ in every institute who knows everyone and everything, we decided to name the system VivaDai. We came up with a huge intertwined system and presented the idea to Talha bhai.
VivaDai 1
Talha bhai immediately turned down our complex system design and proposed a simple file system based memory structure. We implemented exactly that, and the system performed surprisingly well. For what we were aiming, this simple system performed exceptionally well.
But as the conversations and the context grew, the system became increasing unstable and unreliable. If we wanted to make things better, we would need to rewrite.
VivaDai Dos
We then proceeded to rewrite the system from the ground up. Gone was the simple file system based memory. We had a vector store, local embeddings and other fancy features. In total the rewrite took somewhere around 2 weeks. The ‘final‘ system was much better than the inital versions and could manage the memory and context much better.
By this time, we were reporting to Arpa apu (apu is Bangla for sister). We had regular meetings and have her updates constantly. VivaDai Dos was actually deployed well on a server and we also gave a nice demo of the system to apu and Talha bhai.
VivaDai Dos 2
Like any other software project, whenever we thought we were done with a system, the client (Talha bhai in this case) would bring in new ideas and new feature expectations. After the VivaDai Dos demo, Talha bhai wanted new “knowledge visualization“ features.
By this time, Tasdik bhai and many many other people from Bangladesh had come to Nepal for the official office inauguration. It was time for yet another rewrite and Tasdik bhai was joining us on this rewrite. I actually designed a large chunk of the new memory management system for this redesign. The memory was a lot more complicated and now graphs had entered the play field. We worked our assess off for this rewrite. Updated with apu became more and more frequent and we were pushing back deadlines a lot. During our local tests, this new system worked extremely well and managed context almost like a good project manager.
But, during deployment everything fell apart. I don’t want to go into the details, but the demo was extremely extremely bad. As I am writing this post, I do not know what is going to happen to Vivadai.
What’s going to stay with me
There are lots and lots of things I learnt while working on these projects. Here are some of the major things that will stay with me in the long run:
Nothing is complex if you are willing to learn
Before I joined Vivasoft, I had never touched LLMs. Hell, I hadn’t ever tried using the LLM APIs because thought they were “too difficult“. The biggest barrier to my growth was the fact that I made excuses for myself.
When I finally had to work with LLMs and RAG systems, I finally realized how stupid and utterly wrong my prenotions were. Anything can be learnt and anything can be understood if you are willing to put the time into it.
Start Small
One thing that I regularly think about is what would’ve happened if we actually started building that complex intertwined VivaDai system that we initially designed. The only reason we believed in VivaDai and its potential was because we started small and saw what could be done with it.
If we began with the complex system, we would be busy with the messy system and the messy code rather than the larger vision. The project would have been DOA.
Part III : The People
The biggest asset for the Vivasoft are the people of Vivasoft. Throughout my internship I had the privilege of interacting with many people from Bangladesh. Here, I am including some notable moments that helped me form my impression of the company.
We have this Nepali proverb, “एउटा आगो भए, अर्को पानी हुनुपर्छ ।“ (If one person is fire, the other must be water). I have heard this ever since I was a child but I finally fully understood it when I saw the two co-founders of Vivasoft; Shaf bhai and Tareq bhai. Tareq bhai is a soft spoken traditional family man. During the Tihar celebration at the office, Tareq bhai left quite early because the apartment he was staying at closes the gates at 9. This was a huge difference to what happend with Shaf bhai. Shaf bhai took some people from the Nepal office along with Munna bhai and Talha bhai to Thamel and partied the entire night. They are such different people from what I could observe, but somehow their partnership just works.
There’s this demon called Bakasura in the Hindu mythology. He is known for his never ending hunger no matter how much food (or people) he consumes. I feel like Talha bhai is the Bakasura of Vivasoft. He always has a hunger for the next big thing and a hunger for answers and explanations. Utsav, one of my colleagues once said to him “I had a fruitful conversation with you“. Talha bhai comes 10 minutes later and says “What did you mean by a fruitful conversation? Let’s talk about this tomorrow“.
During the early days of the internship, I kind of used to take the company very lightly. I had heard of a few people and interacted only with a handful of people. I only realized about the scale of the company when almost 40 people came to Nepal for the office inauguration and stayed here for about a week. In fact, the office was so jam packed that we had to take our food to the terrace because we had no place to eat.
One of the most notable interactions during this inauguration time was with Alin bhai. I had only heard of him from other people because he “looks like me“. Almost everyday he says “At Vivasoft, we are in the business of creating value“ at least once. See, I am a developer. I only know code and computing systems. I don’t mind if I have to run 7 terminal commands to save a txt file. But, after a good talk with Alin bhai, I think I have a much clearer understanding of what a client is, and what the general user wants. He explains the craziest concepts using Mario metaphors out of all things. One thing that constantly replays in my head is “Your end users are just slightly smart monkeys. They have opposable thumbs but they are stupid at f*ck“.
Tasdik bhai is like a child who already has a sugar rush and then decided to have some Red Bull once again. He is always excited about one thing or another. While he was here, I gave him a lot of shit about him using Windows on his Thinkpad, but honestly he is a genius. He is one of those people who are genuinely excited about working with tech and software. He is very quick with his words and has some obscure reference ready to throw out after each sentence. I expected Tasdik bhai to be the typical millennial who believes everything he believes is right. But, if you have a long enough conversation with him he eventually starts saying “what my parents say“, “my mother said“, and “my father said“.
Conclusion
Congratulations! You have reached the end of this blog post. I hope that this post was helpful to you. If you hope to read more posts like this, consider following me on Hashnode, and joining the newsletter.
If you have any queries or suggestions, please leave a comment or reach out to me directly by sending an email to mukul.development@gmail.com.
