Exploring AI Models: A Journey to Local Implementation
In the ever-evolving landscape of artificial intelligence, numerous models are available for exploration from industry giants like OpenAI and Google, among others. However, engaging with these models often means experiencing them through the lens of the companies that created them, running on their infrastructure. This setup might not appeal to everyone for several reasons. Concerns about data privacy, the desire for customization, and the need for control over the computing environment are just a few motivations for seeking alternatives.
While there are many open AI models available, setting them up for local use can be daunting. It often requires a significant investment in hardware, particularly a powerful video card to serve as a vector processor. Although outsourcing processing is an option, it essentially mirrors the experience of using a hosted chatbot. Fortunately, there are user-friendly solutions for running AI models on personal computers, whether they operate on Windows, Linux, or Mac. One such solution is Msty, a free program for personal use that promises privacy, though users with heightened security concerns may wish to verify this independently.
Introducing Msty
Msty is a versatile desktop application designed to facilitate interaction with AI engines, either locally or remotely. It supports numerous popular options and allows users to input their keys for paid services. For local AI models, Msty can download, install, and run the engines of your choice, offering a seamless experience.
For AI services or engines not directly supported by Msty, users can undertake their own setup, which varies in complexity depending on the specific requirements. While Python or basic interfaces like ollama can be used for local or remote models, Msty enhances the user experience significantly. It allows for file attachments, result exports, and the ability to revisit previous chats. For those who prefer not to retain chat histories, Msty offers a “vapor” mode or the option to delete past interactions.
Each chat session is stored in a dedicated folder, which can include prompts to guide the conversation. For instance, a folder might contain instructions like, “You are an 8th grade math teacher,” setting the stage for the chat.
Engaging with MultiChat
One of Msty’s standout features is its ability to facilitate simultaneous conversations with multiple chatbots. This feature goes beyond mere novelty by allowing users to synchronize chats, prompting each chatbot to respond to the same query. This setup provides a fascinating opportunity to compare the speed and quality of responses from different models.
For example, when asked to explain the workings of a 555 timer, both Google Gemini 2.0 and Llama 3.2 provided distinct answers, highlighting the diversity in AI model capabilities.
Leveraging RAGs for Enhanced Interaction
Msty’s “knowledge stack” feature enables users to compile their own data sources for chat interactions, a process known as Retrieval Augmented Generation (RAG). Users can incorporate files, folders, Obsidian vaults, or YouTube transcripts into their knowledge stack.
For instance, by creating a Knowledge Stack titled “Hackaday Podcast 291” using a YouTube link, users can engage in a chat with Google’s Gemini 2.0 beta, hosted remotely. This setup allows for interactive discussions about the podcast content, such as identifying hosts or discussing specific topics like a probe tip etcher used in atomic-level imaging.
This feature is particularly useful for loading extensive data sets, such as PDF datasheets or design documents, to facilitate detailed project discussions. Additionally, Msty offers a prompt library with a variety of pre-defined roles, from accountants to yogis, for users who prefer not to create their own.
Exploring New AI Models
While Msty’s chat features are impressive, its ability to manage local AI models is perhaps its most valuable asset. The program can download, install, run, and shut down local models with ease.
To begin, users can select the Local AI Model button on the toolbar, which presents several options. It’s important to note that many models are large and require substantial GPU memory. For instance, on a machine equipped with an NVidia 2060 card with 6GB of memory, some smaller models functioned initially but eventually encountered errors. Upgrading to a 12GB 3060 card resolved these issues, allowing for smoother operation, albeit with some larger models running slowly.
Additional options are available by pressing the black button at the top, or users can import GGUF models from sources like huggingface. Msty can also integrate with models already loaded for other platforms like ollama or connect to a local server if preferred.
Although the version tested did not recognize the Google 2.0 model, adding it was straightforward by inputting the (free) API key and model ID (models/gemini-2.0-flash-exp).
Conclusion
Exploring and comparing different AI models can be a time-consuming endeavor. Having a comprehensive list of models is beneficial, though Msty’s pre-existing knowledge base offers a solid starting point.
While Msty is not the only method for running AI models locally, it stands out as one of the most user-friendly options available. Although it is not open source, it is free for personal use, making it an attractive choice for many users. What is your preferred method for running AI? For some, the answer might be to avoid AI altogether, which is a valid perspective as well.
Originally Written by: Al Williams
