New Toy!

My new Arduino Mega

Long time no see folks! I got a new toy this week, an Arduino Mega. Cute lil thing isn’t it? So what is it? The Arduino is a completely open source 8bit RISC controller and board package. Part of the Arduino package is it’s fairly easy to learn library. Since it’s in C++, it’s very easy (for me) to rapidly design little hardware projects. I’ve planned that my first project will be a fan controller for my hard drives. I have two external hard drives and at times, they get a little warm, so I want to write some code for this that communicates with my laptop over USB (serial interface) as to when to turn on the drive fan or not, with manual override of course.

But this is only the surface of what you could do. There are add-ons for the Arduino by a variety of different manufacturers for wifi, ethernet, GPS, bluetooth and even GSM. While the build in Arduino functions have lots of advanced things to play with, you get full control over the processor with the avr-libc library (such as sleep modes). I’m looking forward to seeing what kind of projects I can come up with. Currently, I have two LEDs hooked up via a breadboard that fade back and forth (it has both analog in and out, via PWM) every 30 seconds, or manually if you send it the correct command over serial.

qZDL Alpha on Windows

Compiled with only a little bit of makefile magic needed.
Edit:Yes it does support Windows themes correctly, I just use Windows classic

Git Based Backups, a Followup

I’ve been using git based website backups for a while now. In my previous post, I talked about how my backup system works. So far, it’s worked fairly flawlessly, but there have been lessons learned.

The first lesson I learned is that you CAN dump a MySQL database and keep it in git, however, you have to do it the right way. Turns out, by default, mysqldump puts everything on one line, including new data. This makes for extremely large database dumps, and makes it almost useless to tell what’s new and what’s old. By using “–complete-insert”, it splits each row into a separate SQL command. This makes the initial version quite a bit larger, but subsequent versions smaller. My database 41MB when the dump is complete, which is quite large.

The second lesson I’m now learning is that git stores the entire history locally inside the .git directory. This initially doesn’t take much space, as it’s compressed, but it grows. I have limited space on my host, and currently my .git directory is pushing 131MB. My entire blog directory is now 191MB, so really, the content is just 19MB. That’s a LOT of space (I blame it all on the database, the actual content doesn’t change much), although each individual diff is about 500K. I did some digging, and the kernel.org folk said:

In general, git is not a viable solution for the the case of a large repository with relatively small individual checkouts. However, if developers do not intend to clone, fetch, push into or push from their repositories, then use shallow clones

Well that certainly limits the usefulness now doesn’t it! Unless I’m missing something, it’s basically saying that if you use ‘shallow’ copies, then you can’t really do anything with them unless you create patches locally and apply them in a full copy and commit. In a large team, I would say this makes git almost useless. Back to my situation, it means that it limits the usefulness of backups to almost zero.

So is git the best way to go? Well maybe. Perhaps backing up the database in the method I chose is the best way. I probably would have been better compressing it, naming it, and shipping it to another server. But hey, I had to find this out right?

Git Based Backup

Hey Everybody, sorry about the extended absence. I’ve been very busy of late! I’ve recently decided to try out the newest thing in SCM (Source Control Management), Git. Lots of people are getting on the git bandwagon, and I’m not sure if I’m totally on board yet. One of the things that appealed to me, however, is the ease of setting everything up. I’ve decided, after years, to have an actual backup solution for my website. I’ve been left without backups before, and it’s bitten me in the arse recently. As you can probably guess where this is going, yes, I’m doing git based backups of my website – database and all.

This lets me do some interesting things. First of all, I can have nightly backups, and ‘checkpoints’, points where I could easily roll back changes if I screw something up. Another is a site that only is a snapshot of the live site, whether for testing or redundancy (via branches). And lastly, it lets me have a log of recent changes that I make, and look at changes and have the ability to revert changes that I may have done while working on a script. I typically commit and push changes to a script with a detailed message when I make changes, outside of the nightly backup cycle.

At first, I was curious to see how MySQL dumps would do inside backups. So far they seem to do very well, and don’t mind being in SCM. It gets dumped to text, so stuff that doesn’t change… well doesn’t change. I back up data to one spot regularly, and then semi-frequently to another. This is how it’s laid out:

The live website is denoted with a 1, and the attached database is also pictured. This is where the general traffic and changes are, aka, the critical data. Every night, the data gets replicated to computer 2, which is completely in another country. Unfortunately it’s on a slow link, and not 100% reliable, but hey, at least it’s somewhere right? Computer #3 is my laptop, which is where I may or may not make changes to some files and testing, before committing them. On the server (computer #1), changes are pulled and merged into the repository automatically (if they’re non conflicting), before the fresh changes are committed. I’ve been running this nightly for about a half a week for my low traffic sites, and recently moved it to my blog. The scripts that run will let git automatically add new files, and delete deleted files, having a perfect snapshot of the blog. I’m going to continue testing this on more of my websites, before I make it available to the websites that I host, including a control panel type feature for managing snapshots.

It’s also worth noting that the connection between my server and the main git repository is secured by key based SSH authentication. I haven’t had any issues with this so far and has been an excellent solution.

I like to give credit, where credit is due, so I’m going to give mad props to my webhost, who processed my request for a shell within 15 minutes, installed git within 15 minutes, and fixed the git installation in 15 minutes. Their level of support has been truly remarkable. A big thanks to you guys! Check them out (A Small Orange). If you like them, and decide you want to sign up, don’t forget to use the referral link on my blog .

ZDLSharp/qZDL Preparing for a Beta Release

It’s been some time, but the ZDLSharp Qt beta is nearing beta release. ZDLSharp 3.0.5.3q fixes issues that the C# version of ZDLSharp had. While still in beta, it will be referred to as ZDLSharp, but rebranding will take place once 3.0.5.3q leaves beta to qZDL. It’s currently only available in SVN (See the page for the version for access details), and should compile on all 3 major platforms (Windows, Linux, OSX) without many issues. Bug reporting may be done via the SourceForge Project Page. It’s currently only tested on x86 but there is evidence to support that x64 also works. Testers would be appreciated for all platforms/architectures. To compile, Qt 4.3 or greater is required (Ubuntu 8.04 LTS Hardy Heron or greater for example).

Update: Here’s a screenshot of the DMFlag picker running on Ubuntu 9.10.

qZDL DMFlag Picker

Migrating to a Gentoo-style Update System

Woah woah woah. ZDL is not dead! I’m planning on the new update stuff, and I must say I really like Gentoo’s updating method. There’s two different streams (beta and stable) on at least 3 different operating systems. Here’s a proposed package method…

• qzdl-win-x86 stable unstable
• qzdl-linux-x86 stable unstable
• qzdl-osx-x86 stable unstable

This of course would all be released in qZDL 3.0.6.0, and no plans are being made for x64 or ppc releases. I’m currently working on 3.0.5.3q, which is the transition release between ZDLSharp (C# based) to qZDL (Qt Based). I’m hoping with my past experience with Qt that I can quickly finish up what I have done so far, and start pushing out a release. I’m going to be releasing the ZDLSharp source around that time on my website (it’s currently on request only). Everything should be documented on the wiki, and I just made a commit to the svn repo.

Oh, Hi

Wow, has it really been two months since I’ve blogged? Sorry guys!

If you’ve been talking to me lately, you probably know I’m pretty busy with school. The assignments, papers, and midterms just keep on marching at me as fast as I can do them. Unfortunately, my schedule is extremely inefficient for getting things done. So let’s do a recap!

Christmas

Christmas was pretty busy, and unfortunately the weather played havoc with a few of my plans. There was a few storms around Christmas and on New Years, which forced me to juggle things around a bit. For those that don’t know, I got a shiny new Canon EOS 450D (Rebel XSi), and you can check out pictures on my flickr page.

School

I’m taking a few interesting courses right now:

• System Analysis, Design & Project Management
• Professional Practice
• Computer Organization II
• Operating Systems I
• Anthropology – Religion and the Sacred

This keeps me busy pretty much all the time, and I haven’t had much time to do anything but school work in the past couple weeks.

Programming

All of my current programming project are unfortunately put on hold. Thing’s I’d love to get done is qZDL, and OCam. OCam (name still under consideration), is a little program I developed during the Ottawa OCTranspo strike to look at the City of Ottawa webcams easily. The app lets me flick through all the webcams in a way that’s analogous to using a TV (including “stations”). I’m working to add more and more webcams to this, and maybe release it.

Getting System Information from Linux

Most of the information I gather for my Linux version of my uptime program comes from procfs.

/proc/sys/kernel/hostname

Provides me the one word hostname of this kernel. I’m planning on switching this and /proc/version to a uname() syscall (see man 3 uname) to provide better compatibility on systems without /proc.

/proc/uptime

Provides me the raw number of seconds (including factional) since the computer booted, as well as the total number of seconds this computer has been idle (the second bit I discard)

/proc/loadavg

Provides me with the three load averages

/proc/version

Provides me with a long kernel version, which is different than the one supplied by uname -a.

/proc/meminfo

I get all of my memory information from here, but I don’t use all of the information available to me. Most of it is pretty useless, but I do get:

• Total Memory
• Buffered Memory
• Cached Memory
• Free Memory
• Total Swap
• Free Swap

I get ‘memory used’ from the formula: “int memused = memtotal – cached – buffers – memfree;”

Managing Computer Health in Multiple Places

I have a few computers under my control. Unfortunately, not all of them are in the same room. Actually, most are spread all around the country. I have essentially two computers under my direct control, with a few others that I tend to use.

It becomes very important when you start needing these computers to do tasks, such as serve up websites, provide SSH servers, or other such services. Unfortunately, power interruptions, and crashes do happen. A lot of times, if a server goes down, you need to wait until you can get somebody to physically press the button if it’s remote, or provide some details on network access (did just the network go down, or did the actual computer go down, or both). Collecting information about a server after it has gone down is a trivial task, just look through the logs. The big crunch here is, what about real-time outage information? Having a server “ping” a webpage every 5 minutes, uploading the latest load average, memory information, disk usage, uptime, and system log information should provide adequate information to provide a decent report on system health.

Over the weekend, I want to see if I can write some code and set up a little site where I can monitor all my computers, and maybe even try and send commands to them (security and authentication being somewhat of a problem here).

Update: I started work on this on the weekend, and I already have it working. Visit data.vectec.net/uptime for live uptime information for three of my Linux computers (Mac OSX coming soon I hope).

Poor Man’s Build Farm

Build farms are kinda cool, I deal with one at work. Unfortunately, it’s large, complicated, and very task specific. I did a quick google search, and didn’t find much in alternatives. I’d really like a system which:

• Has an HTTP interface for submitting jobs, getting results, etc
• Allows for any project to be built providing it has a makefile
• Automatic and dynamic resource management
• Relatively simple in implementation usage

The resource management is probably the most curious. Say I’m at a lab computer at school, and I want to use that as a build machine, I would like to be able to run a daemon, which given the right server address and password, will become avilable in the build menu when you submit a job.

There are security problems though. How do you prevent unauthorized users from submitting malicous code? You really can’t, unless you have trusted users. Big companies like SourceForge can afford to create build farms, and probably run the compile in a chrooted environment, but if the disk gets hosed, so what? Just reimage the disk and you’re good to go. Obviously with such distributed systems that really is more of a peer system, that isn’t possible. For example, my own personal laptop could become a build machine.

Of course, ultimately the other option is tarballing the source and emailing it around and getting people to test it on their machines, but that might not that be that easy.

Just a thought.