LLaVA for AI Art Classification July 2024 — August 2024
During my internship at Esperanto Technologies, I had the opportunity to investigate topics in the latest scientific literature for machine learning and develop a project based on this research. After reading LLaVA's foundation paper "Visual Instruction Tuning", I was interested in seeing how the multimodal text-image LLM could perform in a classification task. Using the AI Art dataset ArtBench, I initially generated a dataset of instruction-response pairs by mapping ground truth values to one of 2 predetermined responses and pairing it with a statically defined instruction. Training LLaVA on this dataset garnered an accuracy of 60%. Inspired by the pretraining process described in the model's foundation paper, I constructed a separate dataset of synthetic instruction-response pairs generated by GPT-4 on few-shot prompting. The training pipeline was adjusted to map labels through this dataset. Applying this technique, training the model showed a 73% accuracy in testing across 100 samples.
LLMs, Data Processing, Prompt Engineering, Literature Reviewdemo informal writeup presentation
UR10 Chess Robot November 2023 — April 2024
After a local company donated a UR10 cobot, a portion of the Foothill College Engineering Club was interested in creating an autonomous chess machine. After implementing a simple instruction set for the arm, I developed the pipeline for interaction between URScript, which it runs natively on, and Python. For most of the project, I led development on piece recognition to construct a virtual map of the board, needed for interaction with the chess engine Stockfish. Initial approaches included thresholding the average color value of kernels passed over an input frame for edge detection and convolutional neural networks for classification. To improve compute, we ended up handling this through QR coded pieces and an algorithm which compared an 8x8 binary mask of the current state of the board to an internally stored 8x8 mask of the previous state. This allowed us to determine the state of the board without explicitly labeling the type of each piece.
Computer Vision, IoT, Linear Algebra, Control, Robotics
This project could not have been completed without the talents of Nicholas Glazkov, who implemented much of the control flow and IoT, Thompson Wong, who designed and machined the chassis, and the rest of our gifted team.
repo
.gif)
Various Math Projects July 2023 — December 2023
While taking Differential Equations for the first time, I started thinking about how I could translate what I was learning into code. Revisiting pygame, I first created a solution/flow curve plotter using Heun's Method, a Runge-Kutta method for first-order equations. Building off of this, I developed a direction field plotter using tangents to mimic vectors in a vector field. During my first Linear Algebra course, I implemented, as it appeared then, a novel approach to Gaussian Elimination using Gauss Transforms (p. 221), which was then extended to solve nonsingular, linear systems.
Numerical Methods, Linear Algebra, Differential Equations, Vector Calculussolution plotter direction fields nlsp
Mirroring Arm October 2022 — January 2023
Alongside 3 members of Foothill College's Engineering Club, I led the design and development of a mirroring end effector. Completely new to Arduino, basic circuit design, and computer vision, this project was a big step into the unknown for us as community college freshmen. We handled hand tracking through Google's MediaPipe Solutions computer vision package and mapped the coordinate this would provide us to a rotation of each servo. Serial port access through pySerial allowed us to serialize these rotations for the Arduino. Once the proof of concept was established, a 3D-printed model concluded development of the project.
Computer Vision, Decision Trees, Arduino, 3D Printingnotion
2D Physics Engine June 2020
Developed in Python using the pygame graphics library. At the time, I had little to no experience with physics or trigonometry; most of the learning required to complete this project was outsourced through online forums and the library's documentation.
Simulation, Control Flow, Graphicsrepo