Git Based Backup

Hey Everybody, sorry about the extended absence. I’ve been very busy of late! I’ve recently decided to try out the newest thing in SCM (Source Control Management), Git. Lots of people are getting on the git bandwagon, and I’m not sure if I’m totally on board yet. One of the things that appealed to me, however, is the ease of setting everything up. I’ve decided, after years, to have an actual backup solution for my website. I’ve been left without backups before, and it’s bitten me in the arse recently. As you can probably guess where this is going, yes, I’m doing git based backups of my website – database and all.

This lets me do some interesting things. First of all, I can have nightly backups, and ‘checkpoints’, points where I could easily roll back changes if I screw something up. Another is a site that only is a snapshot of the live site, whether for testing or redundancy (via branches). And lastly, it lets me have a log of recent changes that I make, and look at changes and have the ability to revert changes that I may have done while working on a script. I typically commit and push changes to a script with a detailed message when I make changes, outside of the nightly backup cycle.

At first, I was curious to see how MySQL dumps would do inside backups. So far they seem to do very well, and don’t mind being in SCM. It gets dumped to text, so stuff that doesn’t change… well doesn’t change. I back up data to one spot regularly, and then semi-frequently to another. This is how it’s laid out:

The live website is denoted with a 1, and the attached database is also pictured. This is where the general traffic and changes are, aka, the critical data. Every night, the data gets replicated to computer 2, which is completely in another country. Unfortunately it’s on a slow link, and not 100% reliable, but hey, at least it’s somewhere right? Computer #3 is my laptop, which is where I may or may not make changes to some files and testing, before committing them. On the server (computer #1), changes are pulled and merged into the repository automatically (if they’re non conflicting), before the fresh changes are committed. I’ve been running this nightly for about a half a week for my low traffic sites, and recently moved it to my blog. The scripts that run will let git automatically add new files, and delete deleted files, having a perfect snapshot of the blog. I’m going to continue testing this on more of my websites, before I make it available to the websites that I host, including a control panel type feature for managing snapshots.

It’s also worth noting that the connection between my server and the main git repository is secured by key based SSH authentication. I haven’t had any issues with this so far and has been an excellent solution.

I like to give credit, where credit is due, so I’m going to give mad props to my webhost, who processed my request for a shell within 15 minutes, installed git within 15 minutes, and fixed the git installation in 15 minutes. Their level of support has been truly remarkable. A big thanks to you guys! Check them out (A Small Orange). If you like them, and decide you want to sign up, don’t forget to use the referral link on my blog .

ZDLSharp/qZDL Preparing for a Beta Release

It’s been some time, but the ZDLSharp Qt beta is nearing beta release. ZDLSharp 3.0.5.3q fixes issues that the C# version of ZDLSharp had. While still in beta, it will be referred to as ZDLSharp, but rebranding will take place once 3.0.5.3q leaves beta to qZDL. It’s currently only available in SVN (See the page for the version for access details), and should compile on all 3 major platforms (Windows, Linux, OSX) without many issues. Bug reporting may be done via the SourceForge Project Page. It’s currently only tested on x86 but there is evidence to support that x64 also works. Testers would be appreciated for all platforms/architectures. To compile, Qt 4.3 or greater is required (Ubuntu 8.04 LTS Hardy Heron or greater for example).

Update: Here’s a screenshot of the DMFlag picker running on Ubuntu 9.10.

qZDL DMFlag Picker

Migrating to a Gentoo-style Update System

Woah woah woah. ZDL is not dead! I’m planning on the new update stuff, and I must say I really like Gentoo’s updating method. There’s two different streams (beta and stable) on at least 3 different operating systems. Here’s a proposed package method…

• qzdl-win-x86 stable unstable
• qzdl-linux-x86 stable unstable
• qzdl-osx-x86 stable unstable

This of course would all be released in qZDL 3.0.6.0, and no plans are being made for x64 or ppc releases. I’m currently working on 3.0.5.3q, which is the transition release between ZDLSharp (C# based) to qZDL (Qt Based). I’m hoping with my past experience with Qt that I can quickly finish up what I have done so far, and start pushing out a release. I’m going to be releasing the ZDLSharp source around that time on my website (it’s currently on request only). Everything should be documented on the wiki, and I just made a commit to the svn repo.

Oh, Hi

Wow, has it really been two months since I’ve blogged? Sorry guys!

If you’ve been talking to me lately, you probably know I’m pretty busy with school. The assignments, papers, and midterms just keep on marching at me as fast as I can do them. Unfortunately, my schedule is extremely inefficient for getting things done. So let’s do a recap!

Christmas

Christmas was pretty busy, and unfortunately the weather played havoc with a few of my plans. There was a few storms around Christmas and on New Years, which forced me to juggle things around a bit. For those that don’t know, I got a shiny new Canon EOS 450D (Rebel XSi), and you can check out pictures on my flickr page.

School

I’m taking a few interesting courses right now:

• System Analysis, Design & Project Management
• Professional Practice
• Computer Organization II
• Operating Systems I
• Anthropology – Religion and the Sacred

This keeps me busy pretty much all the time, and I haven’t had much time to do anything but school work in the past couple weeks.

Programming

All of my current programming project are unfortunately put on hold. Thing’s I’d love to get done is qZDL, and OCam. OCam (name still under consideration), is a little program I developed during the Ottawa OCTranspo strike to look at the City of Ottawa webcams easily. The app lets me flick through all the webcams in a way that’s analogous to using a TV (including “stations”). I’m working to add more and more webcams to this, and maybe release it.

Getting System Information from Linux

Most of the information I gather for my Linux version of my uptime program comes from procfs.

/proc/sys/kernel/hostname

Provides me the one word hostname of this kernel. I’m planning on switching this and /proc/version to a uname() syscall (see man 3 uname) to provide better compatibility on systems without /proc.

/proc/uptime

Provides me the raw number of seconds (including factional) since the computer booted, as well as the total number of seconds this computer has been idle (the second bit I discard)

/proc/loadavg

Provides me with the three load averages

/proc/version

Provides me with a long kernel version, which is different than the one supplied by uname -a.

/proc/meminfo

I get all of my memory information from here, but I don’t use all of the information available to me. Most of it is pretty useless, but I do get:

• Total Memory
• Buffered Memory
• Cached Memory
• Free Memory
• Total Swap
• Free Swap

I get ‘memory used’ from the formula: “int memused = memtotal – cached – buffers – memfree;”

Managing Computer Health in Multiple Places

I have a few computers under my control. Unfortunately, not all of them are in the same room. Actually, most are spread all around the country. I have essentially two computers under my direct control, with a few others that I tend to use.

It becomes very important when you start needing these computers to do tasks, such as serve up websites, provide SSH servers, or other such services. Unfortunately, power interruptions, and crashes do happen. A lot of times, if a server goes down, you need to wait until you can get somebody to physically press the button if it’s remote, or provide some details on network access (did just the network go down, or did the actual computer go down, or both). Collecting information about a server after it has gone down is a trivial task, just look through the logs. The big crunch here is, what about real-time outage information? Having a server “ping” a webpage every 5 minutes, uploading the latest load average, memory information, disk usage, uptime, and system log information should provide adequate information to provide a decent report on system health.

Over the weekend, I want to see if I can write some code and set up a little site where I can monitor all my computers, and maybe even try and send commands to them (security and authentication being somewhat of a problem here).

Update: I started work on this on the weekend, and I already have it working. Visit data.vectec.net/uptime for live uptime information for three of my Linux computers (Mac OSX coming soon I hope).

Poor Man’s Build Farm

Build farms are kinda cool, I deal with one at work. Unfortunately, it’s large, complicated, and very task specific. I did a quick google search, and didn’t find much in alternatives. I’d really like a system which:

• Has an HTTP interface for submitting jobs, getting results, etc
• Allows for any project to be built providing it has a makefile
• Automatic and dynamic resource management
• Relatively simple in implementation usage

The resource management is probably the most curious. Say I’m at a lab computer at school, and I want to use that as a build machine, I would like to be able to run a daemon, which given the right server address and password, will become avilable in the build menu when you submit a job.

There are security problems though. How do you prevent unauthorized users from submitting malicous code? You really can’t, unless you have trusted users. Big companies like SourceForge can afford to create build farms, and probably run the compile in a chrooted environment, but if the disk gets hosed, so what? Just reimage the disk and you’re good to go. Obviously with such distributed systems that really is more of a peer system, that isn’t possible. For example, my own personal laptop could become a build machine.

Of course, ultimately the other option is tarballing the source and emailing it around and getting people to test it on their machines, but that might not that be that easy.

Just a thought.

Fibbing Around

My officemate Alexei and I were wondering a bit on how fast Fibonacci in assembly would be, so I wrote a quick little linear fib application in Linux.

08048074 <_start>:
 8048074:       31 c0                   xor    %eax,%eax
 8048076:       bb 01 00 00 00          mov    $0x1,%ebx
 804807b:       ba 01 00 00 00          mov    $0x1,%edx

08048080 <_calc>:
 8048080:       31 c9                   xor    %ecx,%ecx
 8048082:       01 c1                   add    %eax,%ecx
 8048084:       01 d9                   add    %ebx,%ecx
 8048086:       89 d8                   mov    %ebx,%eax
 8048088:       89 cb                   mov    %ecx,%ebx
 804808a:       42                      inc    %edx
 804808b:       83 fa 2e                cmp    $0x2e,%edx
 804808e:       75 f0                   jne    8048080 <_calc>
 8048090:       89 cb                   mov    %ecx,%ebx
 8048092:       b8 01 00 00 00          mov    $0x1,%eax
 8048097:       cd 80                   int    $0x80

Runs pretty quick, but I’m not sure how good it would be, as it’s only using 32 bits, so it can only calculate fib(46) and lower, and doesn’t even output (that’s a bit trickier).

Smallest Hello World

In what turned out to be an assembly vs Java comparison of hello world, I’ve managed to get a hello world app down to a lean 380 bytes. objdump -d:

hello.bin:     file format elf32-i386

Disassembly of section .text:

08048094 <.text>:
 8048094:       ba 0c 00 00 00          mov    $0xc,%edx
 8048099:       b9 b8 90 04 08          mov    $0x80490b8,%ecx
 804809e:       bb 01 00 00 00          mov    $0x1,%ebx
 80480a3:       b8 04 00 00 00          mov    $0x4,%eax
 80480a8:       cd 80                   int    $0x80
 80480aa:       bb 00 00 00 00          mov    $0x0,%ebx
 80480af:       b8 01 00 00 00          mov    $0x1,%eax
 80480b4:       cd 80                   int    $0x80

Can you do better? This hello world does not link to any outside libs like stdlib, and only uses kernel system calls. We were able to get a java 6 compiled class to 517 bytes, but I’m not sure if I could make it smaller (Java bytecode maybe?)

Edit: I found a page on creating really small ELF executables for linux. They don’t say hello world, but it could be easily modified if you knew how to translate AT&T to Intel. I really can’t remember, or have the ambition to do so, so I’ll just link

Application Development Preferences

You’re going to be writing an app. The biggest question is what language will you use?

For some people, it’s a nobrainer, it’s either Java, C++, or other language. What would you use? Usually it has to depend on what the app is for. Lets just say a simple chat client that’s cross platform.

That narrows down the playing field to a few choices:
Java:

• SWT
• AWT
• Swing

CLR (C#, VB, etc):

• Winforms

C/C++

• wxWidgets (GTK)
• Qt

Which would you use, and why? Personally, I like the C/C++ route. I’ve delt with both wxWidgets and Qt, and find both offerings pretty nice. I really like how Qt works over wx, but that’s just personal preference.

Java is getting better these days, but I feel that application start time is still a big issue, as well as memory usage, although these are supposedly getting better. The other issue with Java is that it usually requires people to go and download a JRE before usage, while if you static compile the C/C++ libs, you don’t usually need any extra dependencies (or even just package the dll’s/so’s along with the main executable).

So what about you, oh general audience, what would you choose, and why?