summaryrefslogtreecommitdiffstats
path: root/block/as-iosched.c
Commit message (Collapse)AuthorAgeFilesLines
* as-iosched: properly protect ioc_gone and ioc countJens Axboe2008-07-031-3/+15
| | | | | | | | | | | If we have multiple tasks freeing io contexts when as-iosched is being unloaded, we could complete() ioc_gone twice. Fix that by protecting ioc_gone complete() and clearing with a spinlock for just that purpose. Doesn't matter from a performance perspective, since it'll only enter that path when ioc_gone != NULL (when as-iosched is being rmmod'ed). Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
* block: Fix the starving writes bug in the anticipatory IO schedulerDivyesh Shah2008-07-011-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | AS scheduler alternates between issuing read and write batches. It does the batch switch only after all requests from the previous batch are completed. When switching to a write batch, if there is an on-going read request, it waits for its completion and indicates its intention of switching by setting ad->changed_batch and the new direction but does not update the batch_expire_time for the new write batch which it does in the case of no previous pending requests. On completion of the read request, it sees that we were waiting for the switch and schedules work for kblockd right away and resets the ad->changed_data flag. Now when kblockd enters dispatch_request where it is expected to pick up a write request, it in turn ends the write batch because the batch_expire_timer was not updated and shows the expire timestamp for the previous batch. This results in the write starvation for all the cases where there is the intention for switching to a write batch, but there is a previous in-flight read request and the batch gets reverted to a read_batch right away. This also holds true in the reverse case (switching from a write batch to a read batch with an in-flight write request). I've checked that this bug exists on 2.6.11, 2.6.18, 2.6.24 and linux-2.6-block git HEAD. I've tested the fix on x86 platforms with SCSI drives where the driver asks for the next request while a current request is in-flight. This patch is based off linux-2.6-block git HEAD. Bug reproduction: A simple scenario which reproduces this bug is: - dd if=/dev/hda3 of=/dev/null & - lilo The lilo takes forever to complete. This can also be reproduced fairly easily with the earlier dd and another test program doing msync(). The example test program below should print out a message after every iteration but it simply hangs forever. With this bugfix it makes forward progress. ==== Example test program using msync() (thanks to suleiman AT google DOT com) inline uint64_t rdtsc(void) { int64_t tsc; __asm __volatile("rdtsc" : "=A" (tsc)); return (tsc); } int main(int argc, char **argv) { struct stat st; uint64_t e, s, t; char *p, q; long i; int fd; if (argc < 2) { printf("Usage: %s <file>\n", argv[0]); return (1); } if ((fd = open(argv[1], O_RDWR | O_NOATIME)) < 0) err(1, "open"); if (fstat(fd, &st) < 0) err(1, "fstat"); p = mmap(NULL, st.st_size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); t = 0; for (i = 0; i < 1000; i++) { *p = 0; msync(p, 4096, MS_SYNC); s = rdtsc(); *p = 0; __asm __volatile(""::: "memory"); e = rdtsc(); if (argc > 2) printf("%d: %lld cycles %jd %jd\n", i, e - s, (intmax_t)s, (intmax_t)e); t += e - s; } printf("average time: %lld cycles\n", t / 1000); return (0); } Cc: <stable@kernel.org> Acked-by: Nick Piggin <npiggin@suse.de> Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
* block: kill swap_io_context()Jens Axboe2008-02-011-14/+0
| | | | | | | | | | It blindly copies everything in the io_context, including the lock. That doesn't work so well for either lock ordering or lockdep. There seems zero point in swapping io contexts on a request to request merge, so the best point of action is to just remove it. Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
* as-iosched: fix inconsistent ioc->lock contextJens Axboe2008-02-011-4/+6
| | | | | | | | | | | Since it's acquired from irq context, all locking must be of the irq safe variant. Most are already inside the queue lock (which already disables interrupts), but the io scheduler rmmod path always has irqs enabled and the put_io_context() path may legally be called with irqs enabled (even if it isn't usually). So fixup those two. Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
* as-iosched: fix double locking bug in as_merged_requests()Jens Axboe2008-01-301-3/+7
| | | | | | | | | | If the two requests belong to the same io context, we will attempt to lock the same lock twice. But swapping contexts is pointless in that case, so just check for rioc == nioc before doing the double lock and copy. Tested-by: Olof Johansson <olof@lixom.net> Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
* io_context sharing - anticipatory changesJens Axboe2008-01-281-6/+28
| | | | | | changes to anticipatory io scheduler for io_context sharing Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
* block: let elv_register() return voidAdrian Bunk2007-12-181-1/+3
| | | | | | | | | elv_register() always returns 0, and there isn't anything it does where it should return an error (the only error condition is so grave that it's handled with a BUG_ON). Signed-off-by: Adrian Bunk <bunk@kernel.org> Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
* as-iosched: fix write batch start pointAaron Carroll2007-12-181-1/+2
| | | | | | | | | | | | New write batches currently start from where the last one completed. We have no idea where the head is after switching batches, so this makes little sense. Instead, start the next batch from the request with the earliest deadline in the hope that we avoid a deadline expiry later on. Signed-off-by: Aaron Carroll <aaronc@gelato.unsw.edu.au> Acked-by: Nick Piggin <npiggin@suse.de> Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
* as-iosched: fix incorrect commentsAaron Carroll2007-12-181-2/+2
| | | | | | | | | Two comments refer to deadlines applying to reads only. This is not the case. Signed-off-by: Aaron Carroll <aaronc@gelato.unsw.edu.au> Acked-by: Nick Piggin <npiggin@suse.de> Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
* [BLOCK] Get rid of request_queue_t typedefJens Axboe2007-07-241-12/+14
| | | | | | | | | Some of the code has been gradually transitioned to using the proper struct request_queue, but there's lots left. So do a full sweet of the kernel and get rid of this typedef and replace its uses with the proper type. Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
* Slab allocators: Replace explicit zeroing with __GFP_ZEROChristoph Lameter2007-07-171-2/+1
| | | | | | | | | | | | | kmalloc_node() and kmem_cache_alloc_node() were not available in a zeroing variant in the past. But with __GFP_ZERO it is possible now to do zeroing while allocating. Use __GFP_ZERO to remove the explicit clearing of memory via memset whereever we can. Signed-off-by: Christoph Lameter <clameter@sgi.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
* kblockd: use flush_workAndrew Morton2007-05-091-1/+1
| | | | | | | | | | | | | | | | Switch the kblockd flushing from a global flush to a more specific flush_work(). (akpm: bypassed maintainers, sorry. There are other patches which depend on this) Cc: "Maciej W. Rozycki" <macro@linux-mips.org> Cc: David Howells <dhowells@redhat.com> Cc: Jens Axboe <axboe@suse.de> Cc: Nick Piggin <nickpiggin@yahoo.com.au> Cc: Oleg Nesterov <oleg@tv-sign.ru> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
* as: fix antic_expire checkNick Piggin2007-05-081-3/+3
| | | | | | | | | | Fix units mismatch (jiffies vs msecs) in as-iosched.c, spotted by Xiaoning Ding <dingxn@cse.ohio-state.edu>. Signed-off-by: Nick Piggin <npiggin@suse.de> Cc: Jens Axboe <jens.axboe@oracle.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
* [PATCH] Allow as-iosched to be unloadedJens Axboe2006-12-131-14/+1
| | | | | | | | We implemented the missing bits to allow this some time ago, and they are integrated in AS. So remove the __module_get() to allow the module to be unloaded. Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
* Merge branch 'master' of ↵David Howells2006-12-051-1/+1
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6 Conflicts: drivers/infiniband/core/iwcm.c drivers/net/chelsio/cxgb2.c drivers/net/wireless/bcm43xx/bcm43xx_main.c drivers/net/wireless/prism54/islpci_eth.c drivers/usb/core/hub.h drivers/usb/input/hid-core.c net/core/netpoll.c Fix up merge failures with Linus's head and fix new compilation failures. Signed-Off-By: David Howells <dhowells@redhat.com>
| * [BLOCK] Cleanup unused variable passingJens Axboe2006-12-011-1/+1
| | | | | | | | | | | | | | | | - ->init_queue() does not need the elevator passed in - ->put_request() is a hot path and need not have the queue passed in - cfq_update_io_seektime() does not need cfqd passed in Signed-off-by: Jens Axboe <jens.axboe@oracle.com>
* | WorkStruct: Pass the work_struct pointer instead of context dataDavid Howells2006-11-221-3/+4
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Pass the work_struct pointer to the work function rather than context data. The work function can use container_of() to work out the data. For the cases where the container of the work_struct may go away the moment the pending bit is cleared, it is made possible to defer the release of the structure by deferring the clearing of the pending bit. To make this work, an extra flag is introduced into the management side of the work_struct. This governs auto-release of the structure upon execution. Ordinarily, the work queue executor would release the work_struct for further scheduling or deallocation by clearing the pending bit prior to jumping to the work function. This means that, unless the driver makes some guarantee itself that the work_struct won't go away, the work function may not access anything else in the work_struct or its container lest they be deallocated.. This is a problem if the auxiliary data is taken away (as done by the last patch). However, if the pending bit is *not* cleared before jumping to the work function, then the work function *may* access the work_struct and its container with no problems. But then the work function must itself release the work_struct by calling work_release(). In most cases, automatic release is fine, so this is the default. Special initiators exist for the non-auto-release case (ending in _NAR). Signed-Off-By: David Howells <dhowells@redhat.com>
* [PATCH] completions: lockdep annotate on stack completionsPeter Zijlstra2006-10-011-1/+1
| | | | | | | | | | | All on stack DECLARE_COMPLETIONs should be replaced by: DECLARE_COMPLETION_ONSTACK Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Acked-by: Ingo Molnar <mingo@elte.hu> Acked-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Andrew Morton <akpm@osdl.org> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
* [PATCH] Update axboe@suse.de email addressJens Axboe2006-09-301-1/+1
| | | | | | | As people often look for the copyright in files to see who to mail, update the link to a neutral one. Signed-off-by: Jens Axboe <axboe@kernel.dk>
* [PATCH] Add blk_start_queueing() helperJens Axboe2006-09-301-2/+1
| | | | | | | | | | CFQ implements this on its own now, but it's really block layer knowledge. Tells a device queue to start dispatching requests to the driver, taking care to unplug if needed. Also fixes the issue where as/cfq will invoke a stopped queue, which we really don't want. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] Make sure all block/io scheduler setups are node awareJens Axboe2006-09-301-4/+4
| | | | | | | Some were kmalloc_node(), some were still kmalloc(). Change them all to kmalloc_node(). Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] as-iosched: use new io context counting mechanismJens Axboe2006-09-301-4/+5
| | | | | | | | It's ok if the read path is a lot more costly, as long as inc/dec is really cheap. The inc/dec will happen for each created/freed io context, while the reading only happens when a disk queue exits. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] Drop useless bio passing in may_queue/set_request APIJens Axboe2006-09-301-1/+1
| | | | | | It's not needed for anything, so kill the bio passing. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] as-iosched: kill arqJens Axboe2006-09-301-195/+118
| | | | | | | | | Get rid of the as_rq request type. With the added elevator_private2, we have enough room in struct request to get rid of any arq allocation/free for each request. Signed-off-by: Jens Axboe <axboe@suse.de> Signed-off-by: Nick Piggin <npiggin@suse.de>
* [PATCH] as-iosched: remove arq->is_sync memberJens Axboe2006-09-301-22/+14
| | | | | | | We can track this in struct request. Signed-off-by: Jens Axboe <axboe@suse.de> Signed-off-by: Nick Piggin <npiggin@suse.de>
* [PATCH] as-iosched: reuse rq for fifoJens Axboe2006-09-301-22/+12
| | | | | | | Saves some space in arq. Signed-off-by: Jens Axboe <axboe@suse.de> Signed-off-by: Nick Piggin <npiggin@suse.de>
* [PATCH] as-iosched: migrate to using the elevator rb functionsJens Axboe2006-09-301-151/+29
| | | | | | | This removes the rbtree handling from AS. Signed-off-by: Jens Axboe <axboe@suse.de> Signed-off-by: Nick Piggin <npiggin@suse.de>
* [PATCH] rbtree: fixed reversed RB_EMPTY_NODE and rb_next/prevJens Axboe2006-09-301-2/+2
| | | | | | | The conditions got reserved. Also make rb_next() and rb_prev() check for the empty condition. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] elevator: move the backmerging logic into the elevator coreJens Axboe2006-09-301-136/+3
| | | | | | | | | | | Right now, every IO scheduler implements its own backmerging (except for noop, which does no merging). That results in duplicated code for essentially the same operation, which is never a good thing. This patch moves the backmerging out of the io schedulers and into the elevator core. We save 1.6kb of text and as a bonus get backmerging for noop as well. Win-win! Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] Split struct request ->flags into two partsJens Axboe2006-09-301-1/+1
| | | | | | | | | | Right now ->flags is a bit of a mess: some are request types, and others are just modifiers. Clean this up by splitting it into ->cmd_type and ->cmd_flags. This allows introduction of generic Linux block message types, useful for sending generic Linux commands to block devices. Signed-off-by: Jens Axboe <axboe@suse.de>
* Remove obsolete #include <linux/config.h>Jörn Engel2006-06-301-1/+0
| | | | | Signed-off-by: Jörn Engel <joern@wohnheim.fh-wedel.de> Signed-off-by: Adrian Bunk <bunk@stusta.de>
* spelling fixesAndreas Mohr2006-06-261-1/+1
| | | | | | | | | | | | acquired (aquired) contiguous (contigious) successful (succesful, succesfull) surprise (suprise) whether (weather) some other misspellings Signed-off-by: Andreas Mohr <andi@lisas.de> Signed-off-by: Adrian Bunk <bunk@stusta.de>
* [PATCH] rbtree: support functions used by the io schedulersJens Axboe2006-06-231-10/+7
| | | | | | | They all duplicate macros to check for empty root and/or node, and clearing a node. So put those in rbtree.h. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] Kill PF_SYNCWRITE flagJens Axboe2006-06-231-1/+1
| | | | | | | | | | | | | | | A process flag to indicate whether we are doing sync io is incredibly ugly. It also causes performance problems when one does a lot of async io and then proceeds to sync it. Part of the io will go out as async, and the other part as sync. This causes a disconnect between the previously submitted io and the synced io. For io schedulers such as CFQ, this will cause us lost merges and suboptimal behaviour in scheduling. Remove PF_SYNCWRITE completely from the fsync/msync paths, and let the O_DIRECT path just directly indicate that the writes are sync by using WRITE_SYNC instead. Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] iosched: use hlist for request hashtableAkinobu Mita2006-06-231-26/+19
| | | | | | | | | | | | Use hlist instead of list_head for request hashtable in deadline-iosched and as-iosched. It also can remove the flag to know hashed or unhashed. Signed-off-by: Akinobu Mita <mita@miraclelinux.com> Signed-off-by: Jens Axboe <axboe@suse.de> block/as-iosched.c | 45 +++++++++++++++++++-------------------------- block/deadline-iosched.c | 39 ++++++++++++++++----------------------- 2 files changed, 35 insertions(+), 49 deletions(-)
* Merge git://git.infradead.org/~dwmw2/rbtree-2.6Linus Torvalds2006-06-201-3/+2
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * git://git.infradead.org/~dwmw2/rbtree-2.6: [RBTREE] Switch rb_colour() et al to en_US spelling of 'color' for consistency Update UML kernel/physmem.c to use rb_parent() accessor macro [RBTREE] Update hrtimers to use rb_parent() accessor macro. [RBTREE] Add explicit alignment to sizeof(long) for struct rb_node. [RBTREE] Merge colour and parent fields of struct rb_node. [RBTREE] Remove dead code in rb_erase() [RBTREE] Update JFFS2 to use rb_parent() accessor macro. [RBTREE] Update eventpoll.c to use rb_parent() accessor macro. [RBTREE] Update key.c to use rb_parent() accessor macro. [RBTREE] Update ext3 to use rb_parent() accessor macro. [RBTREE] Change rbtree off-tree marking in I/O schedulers. [RBTREE] Add accessor macros for colour and parent fields of rb_node
| * [RBTREE] Change rbtree off-tree marking in I/O schedulers.David Woodhouse2006-04-211-3/+2
| | | | | | | | | | | | | | | | | | They were abusing the rb_color field to mark nodes which weren't currently on the tree. Fix that to use the same method as eventpoll did -- setting the parent pointer to point back to itself. And use the appropriate accessor macros for setting and reading the parent. Signed-off-by: David Woodhouse <dwmw2@infradead.org>
* | [PATCH] elevator switching raceJens Axboe2006-06-081-7/+6
|/ | | | | | | | | | | | | | | | | | | | There's a race between shutting down one io scheduler and firing up the next, in which a new io could enter and cause the io scheduler to be invoked with bad or NULL data. To fix this, we need to maintain the queue lock for a bit longer. Unfortunately we cannot do that, since the elevator init requires to be run without the lock held. This isn't easily fixable, without also changing the mempool API. So split the initialization into two parts, and alloc-init operation and an attach operation. Then we can preallocate the io scheduler and related structures, and run the attach inside the lock after we detach the old one. This patch has survived 30 minutes of 1 second io scheduler switching with a very busy io load. Signed-off-by: Jens Axboe <axboe@suse.de> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
* [PATCH 1/2] iosched: fix typo and barrier()OGAWA Hirofumi2006-04-181-2/+3
| | | | | | | | | | | | | | | | | | | | | | On rmmod path, cfq/as waits to make sure all io-contexts was freed. However, it's using complete(), not wait_for_completion(). I think barrier() is not enough in here. To avoid the following case, this patch replaces barrier() with smb_wmb(). cpu0 visibility cpu1 [ioc_gnone=NULL,ioc_count=1] ioc_gnone = &all_gone NULL,ioc_count=1 atomic_read(&ioc_count) NULL,ioc_count=1 wait_for_completion() NULL,ioc_count=0 atomic_sub_and_test() NULL,ioc_count=0 if ( && ioc_gone) [ioc_gone==NULL, so doesn't call complete()] &all_gone,ioc_count=0 Signed-off-by: OGAWA Hirofumi <hirofumi@mail.parknet.co.jp> Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] fix rmmod problems with elevator attributes, clean them upAl Viro2006-03-181-49/+22
|
* [PATCH] elevator_t lifetime rules and sysfs fixesAl Viro2006-03-181-54/+15
|
* [PATCH] deal with rmmod/put_io_context() racesAl Viro2006-03-181-1/+14
| | | | Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
* [PATCH] stop elv_unregister() from rogering other iosched's data, fix lockingAl Viro2006-03-181-0/+7
| | | | Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
* [BLOCK] as-iosched: update alias handlingTejun Heo2006-01-061-119/+25
| | | | | | | | | | | Unlike other ioscheds, as-iosched handles alias by chaing them using rq->queuelist. As aliased requests are very rare in the first place, this complicates merge/dispatch handling without meaningful performance improvement. This patch updates as-iosched to dump aliased requests into dispatch queue as other ioscheds do. Signed-off-by: Tejun Heo <htejun@gmail.com> Signed-off-by: Jens Axboe <axboe@suse.de>
* [PATCH] as-iosched: remove state assertion in as_add_request()Jens Axboe2005-11-211-4/+0
| | | | | | | | | | | Kill the arq->state poison statement in as_add_request(), it can trigger for perfectly valid code that just reuses a request after io completion instead of freeing it and allocating a new one. We probably should introduce a blk_init_request() to start from scratch, but for now just kill it as we will be removing the as specific poisoning soon. Signed-off-by: Jens Axboe <axboe@suse.de> Signed-off-by: Linus Torvalds <torvalds@osdl.org>
* [BLOCK] new block/ directory comment tidyCoywolf Qi Hunt2005-11-181-2/+0
| | | | | | | | Some leftover comments referring to drivers/block that are now block/. They don't add any information we don't already have, so kill them. Signed-off-by: Coywolf Qi Hunt <qiyong@fc-cn.com> Signed-off-by: Jens Axboe <axboe@suse.de>
* Merge branch 'block-dir' of git://brick.kernel.dk/data/git/linux-2.6-blockLinus Torvalds2005-11-071-145/+165
|
* [BLOCK] Move all core block layer code to new block/ directoryJens Axboe2005-11-041-0/+1985
drivers/block/ is right now a mix of core and driver parts. Lets move the core parts to a new top level directory. Al will move the fs/ related block parts to block/ next. Signed-off-by: Jens Axboe <axboe@suse.de>
OpenPOWER on IntegriCloud