Category: Linux

Speed up a new VPS

Reducing system logging activity

In a default distro install, system logging is often configured fully, suitable for a server or multi-user system. However, on a single-user system the constant writing the many system log files will result in reduced interactive system performance, and reducing logging activity will be beneficial for performance as well as the lifetime of the flash memory.

In a Debian Wheezy installation, the default system logger is rsyslog, and it configuration file is /etc/rsyslog.conf. In the rules section, the following logs are often enabled by default:

auth,authpriv.*                        /var/log/auth.log
*.*;auth,authpriv.none         -/var/log/syslog
cron.*                         /var/log/cron.log
daemon.*                       -/var/log/daemon.log
kern.*                         -/var/log/kern.log
lpr.*                          -/var/log/lpr.log
mail.*                         -/var/log/mail.log
user.*                         -/var/log/user.log

There may also be rules for -/var/log/debug and -/var/log/messages, and |/dev/xconsole.

Note that kernel messages are logged in both kern.log and syslog, in addition to being available from the dmesg command from kernel memory. In a single user system, it is possible to disable most or all of these logs by placing a ‘#’ character at the start of the corresponding lines. Logs can be re-enabled if it is necessary to debug a system problem.

Special flags for your mounts

By default, many distributions including Ubuntu use the ‘relatime’ flag for updating file metadata when file are accesed, but if you’re unlikely to care about last access time you can skip this. Indeed this will come with both improve performance and, more importantly, the longevity of your SSD by reducing unnecessary writes.

To make all these changes, open up a terminal and run:

sudo nano -w /etc/fstab

Then for all SSD devices in your system remove ‘relatime’ if present and add ‘noatime’ so it looks something like this:

/dev/sdaX   /   ext4   defaults,noatime,errors=remount-ro 0 1
/dev/sdaY   /home   ext4   defaults,noatime,errors=remount-ro 0 2

As you can see I didn’t use the nodiratime nor discard. In the first one the usage of noatime has nodiratime implicit. And in the second one I have experimented some performance drawbacks when performing operations with large number of small files.

If it’s temporary move it to RAM

Every day applications generat a lot of log files so to reduce unnecessary writes to disk move the temp directories into a ram disk using the ‘tmpfs’ filesystem, which dynamically expands and shrinks as needed.

In your /etc/fstab, add the following:

tmpfs   /tmp       tmpfs   defaults,noatime,mode=1777   0  0
tmpfs   /var/spool tmpfs   defaults,noatime,mode=1777   0  0
tmpfs   /var/tmp   tmpfs   defaults,noatime,mode=1777   0  0

If you don’t mind losing log files between boots, and unless you’re running a server you can probably live without them, also add:

tmpfs   /var/log   tmpfs   defaults,noatime,mode=0755   0  0

Optimizing cache directories

Applications like browsers and window managers that use a disk cache may conform to the XDG Base Directory Specification standard. In that case, the environment variable XDG_CACHE_HOME defines the directory where cache files are stored. By setting this variable to a ramdisk location, it is possible to greatly speed-up the performance of certain browsers that otherwise stall with heavy writing to the disk-cache in flash memory. For example, in a Debian Wheezy installation with LXDE, there may be a configuration file called /etc/alternatives/x-session-manager. By adding the line

export XDG_CACHE_HOME="/dev/shm/.cache"

to the start of this file, programs running in X conforming to the standard will be using the ramdisk in /dev/shm to store cache files. One browser popular on light-weight configurations that benefits form this is Midori. The default disk cache size in Midori is 100MB, you can lower

Better Linux Disk Caching & Performance with vm.dirty_ratio & vm.dirty_background_ratio

File caching is an important performance improvement, and read caching is a clear win in most cases, balanced against applications using the RAM directly. Write caching is trickier. The Linux kernel stages disk writes into cache, and over time asynchronously flushes them to disk. This has a nice effect of speeding disk I/O but it is risky. When data isn’t written to disk there is an increased chance of losing it.

There is also the chance that a lot of I/O will overwhelm the cache, too. Ever written a lot of data to disk all at once, and seen large pauses on the system while it tries to deal with all that data? Those pauses are a result of the cache deciding that there’s too much data to be written asynchronously (as a non-blocking background operation, letting the application process continue), and switches to writing synchronously (blocking and making the process wait until the I/O is committed to disk). Of course, a filesystem also has to preserve write order, so when it starts writing synchronously it first has to destage the cache. Hence the long pause.

The nice thing is that these are controllable options, and based on your workloads & data you can decide how you want to set them up. Let’s take a look:

$ sysctl -a | grep dirty
 vm.dirty_background_ratio = 10
 vm.dirty_background_bytes = 0
 vm.dirty_ratio = 20
 vm.dirty_bytes = 0
 vm.dirty_writeback_centisecs = 500
 vm.dirty_expire_centisecs = 3000

vm.dirty_background_ratio is the percentage of system memory that can be filled with “dirty” pages — memory pages that still need to be written to disk — before the pdflush/flush/kdmflush background processes kick in to write it to disk. My example is 10%, so if my virtual server has 32 GB of memory that’s 3.2 GB of data that can be sitting in RAM before something is done.

vm.dirty_ratio is the absolute maximum amount of system memory that can be filled with dirty pages before everything must get committed to disk. When the system gets to this point all new I/O blocks until dirty pages have been written to disk. This is often the source of long I/O pauses, but is a safeguard against too much data being cached unsafely in memory.

vm.dirty_background_bytes and vm.dirty_bytes are another way to specify these parameters. If you set the _bytes version the _ratio version will become 0, and vice-versa.

vm.dirty_expire_centisecs is how long something can be in cache before it needs to be written. In this case it’s 30 seconds. When the pdflush/flush/kdmflush processes kick in they will check to see how old a dirty page is, and if it’s older than this value it’ll be written asynchronously to disk. Since holding a dirty page in memory is unsafe this is also a safeguard against data loss.

vm.dirty_writeback_centisecs is how often the pdflush/flush/kdmflush processes wake up and check to see if work needs to be done.

There are also scenarios where a system has to deal with infrequent, bursty traffic to slow disk (batch jobs at the top of the hour, midnight, writing to an SD card on a Raspberry Pi, etc.). In that case an approach might be to allow all that write I/O to be deposited in the cache so that the background flush operations can deal with it asynchronously over time:

vm.dirty_background_ratio = 5
vm.dirty_ratio = 80

Here the background processes will start writing right away when it hits that 5% ceiling but the system won’t force synchronous I/O until it gets to 80% full. From there you just size your system RAM and vm.dirty_ratio to be able to consume all the written data. Again, there are tradeoffs with data consistency on disk, which translates into risk to data. Buy a UPS and make sure you can destage cache before the UPS runs out of power. icon smile Speed up a new  VPS

No matter the route you choose you should always be gathering hard data to support your changes and help you determine if you are improving things or making them worse. In this case you can get data from many different places, including the application itself, /proc/vmstat, /proc/meminfo, iostat, vmstat, and many of the things in /proc/sys/vm.

Check if swap is enabled on your VPS, if not create it

The “free” command shows your system’s available physical and virtual memory.

If you have virtual memory enabled already, you can skip ahead to “A Note About Swap Partitions” and then the configuration section. When enabled, the output will look like this: free
             total       used       free     shared    buffers     cached
Mem:        361996     360392       1604          0       1988      54376
-/+ buffers/cache:     304028      57968
Swap:       249896          0     249896 _

If it is not enabled, the output will look like this: free
             total       used       free     shared    buffers     cached
Mem:        361996     360392       1604          0       2320      54444
-/+ buffers/cache:     303628      58368
Swap:            0          0          0 _

You can also narrow down the output with free | grep Swap. This will only show theSwap: line, total, used and free VM. (Remember, by default, grep is case sensitive!) free | grep Swap
Swap:       249896          0     249896 _


Virtual memory allows your system (and thus your apps) additional virtual RAM beyond what your system physically has – or in the case of droplets, what is allocated. It does this by using your disk for the extra, ‘virtual’ memory and swaps data in and out of system memory and virtual memory as it’s needed.

to create and use it

root@moscheni:~# dd if=/dev/zero of=/var/swap.img bs=1024k count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 7.45802 s, 281 MB/s
root@moscheni:~# mkswap /var/swap.img 
Setting up swapspace version 1, size = 2047996 KiB
no label, UUID=5331c54f-d407-40c1-9eb9-a019cacb6ee8
root@moscheni:~# swapo
swapoff swapon 
root@moscheni:~# swapon /var/swap.img 
root@moscheni:~# free -m 
 total used free shared buffers cached
Mem: 5988 2126 3861 0 5 2017
-/+ buffers/cache: 103 5884
Swap: 1999 0 1999

I preferer from 512MB to 2GB





share save 120 16 Speed up a new  VPS

Installare Debian GNU/kFreeBSD con supporto ZFS

Cos’è Debian GNU/kFreeBSD ?
Debian GNU/kFreeBSD è un sistema operativo per Debian, che usa il kernel di FreeBSD invece del kernel Linux (da qui il nome). Verrà il giorno in cui la maggior parte delle applicazioni esisterà sia in Debian GNU/kFreeBSD sia in Debian GNU/Linux.

Cos’è ZFS?
ZFS è un nuovo tipo di file system che fornisce una semplice amministrazione, una approccio transazionale, una integrità del dato “end-to-end” e una immensa scalabilità (128-bit). ZFS non è un miglioramento “incrementale” alla tecnologia esistente, ma è un nuovo approccio alla gestione dei dati creato eliminando alcune assunzioni di base che risalevano a 20 anni fa.

Come si installa?

1) download della iso solo con supporto amd64 di debian kfreebsd per esempio cosi
curl -C – -L -O

2) far partire l’installazione di defualt (si sono pigro)
Selezione 0021 300x171 Installare Debian GNU/kFreeBSD con supporto ZFS
3) seguire i passi per lingua , mirror , configurazione di rete , user e password, nel virtual machine allegata ( user root password ferzip , e utente user con password ferzip )
Selezione 0031 300x171 Installare Debian GNU/kFreeBSD con supporto ZFS

4) attenzione in macchine virtuali durante l’installazione potrebbe presentarsi un errore di di svuotamento cache del disco
Selezione 0041 300x172 Installare Debian GNU/kFreeBSD con supporto ZFS

5) procedere poi con la formattazione del disco , attenzione se si usa la procedura guidata personalizzare la partizione selezionando il filesistem ZFS come da slide qua sotto

6) procedere con la normale installazione e quando richiesto selezione l’evemenent prescelto
Selezione 014 300x171 Installare Debian GNU/kFreeBSD con supporto ZFS

7) continuare con la procedura guidata fino alle fine
Selezione 016 300x171 Installare Debian GNU/kFreeBSD con supporto ZFS

8) TATA ecco il nuovo sistema installato
user@ferzip-freeBSD-zfs:~$ uname -a
GNU/kFreeBSD ferzip-freeBSD-zfs 8.1-1-amd64 #0 Wed Oct 19 14:57:54 CEST 2011 x86_64 amd64 Intel(R) Core(TM) i3 CPU M 350 @ 2.27GHz GNU/kFreeBSD
user@ferzip-freeBSD-zfs:~$ mount
ferzip-freeBSD-zfs-ad0s1 on / (zfs, local)
devfs on /dev (devfs, local, multilabel)
linprocfs on /proc (linprocfs, local)
linsysfs on /sys (linsysfs, local)
fdescfs on /dev/fd (fdescfs)
tmpfs on /lib/init/rw (tmpfs, local, nosuid)

share save 120 16 Installare Debian GNU/kFreeBSD con supporto ZFS

How to find open proxy

I wrote a simple shell script to find a open proxy, basicaly it analize the log of webserver , extract the source ip client and try if on this ip have a proxy( only some standard port 80,8080,3128).This script don’t have “memory” if in log find more than one istance of ip, retry the check every time.
Only the ip who running an open proxy is strored in a file call “result.log”, to run this script yun need only curl.

share save 120 16 How to find open proxy

Usare periferiche USB con QEMU

Sto facendo usando QEMU dal mio sistema Debian quando ho la necessita di attaccare una dispositivo USB all mia macchina virtuale, molto semplicemente passo il parametro “-usb” a qemu:

Prima identifico con il comando lsusb la periferica

Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 002: ID 413c:3016 Dell Computer Corp. Optical 5-Button Wheel Mouse
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 004: ID 0781:5151 SanDisk Corp. Cruzer Micro Flash Drive
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

e poi semplicemente passo i dati a qemu con i parametri necessari

sudo qemu /tmp/xp.img -std-vga -m 1024 -usb -usbdevice host:0781:5151

Per avere il pieno supporto all’usb bisogno avere l’accesso come root

share save 120 16 Usare periferiche USB con QEMU
Get Adobe Flash player