From 2c14a6831954a59256cc8d1980da0ad705a3a3fa Mon Sep 17 00:00:00 2001 From: Juan Quintela <quintela@redhat.com> Date: Tue, 3 Mar 2020 14:51:37 +0000 Subject: [PATCH 05/18] migration/multifd: fix destroyed mutex access in terminating multifd threads RH-Author: Juan Quintela <quintela@redhat.com> Message-id: <20200303145143.149290-5-quintela@redhat.com> Patchwork-id: 94119 O-Subject: [RHEL-AV-8.2.0 qemu-kvm PATCH v2 04/10] migration/multifd: fix destroyed mutex access in terminating multifd threads Bugzilla: 1738451 RH-Acked-by: Laurent Vivier <lvivier@redhat.com> RH-Acked-by: Peter Xu <peterx@redhat.com> RH-Acked-by: Dr. David Alan Gilbert <dgilbert@redhat.com> From: Jiahui Cen <cenjiahui@huawei.com> One multifd will lock all the other multifds' IOChannel mutex to inform them to quit by setting p->quit or shutting down p->c. In this senario, if some multifds had already been terminated and multifd_load_cleanup/multifd_save_cleanup had destroyed their mutex, it could cause destroyed mutex access when trying lock their mutex. Here is the coredump stack: #0 0x00007f81a2794437 in raise () from /usr/lib64/libc.so.6 #1 0x00007f81a2795b28 in abort () from /usr/lib64/libc.so.6 #2 0x00007f81a278d1b6 in __assert_fail_base () from /usr/lib64/libc.so.6 #3 0x00007f81a278d262 in __assert_fail () from /usr/lib64/libc.so.6 #4 0x000055eb1bfadbd3 in qemu_mutex_lock_impl (mutex=0x55eb1e2d1988, file=<optimized out>, line=<optimized out>) at util/qemu-thread-posix.c:64 #5 0x000055eb1bb4564a in multifd_send_terminate_threads (err=<optimized out>) at migration/ram.c:1015 #6 0x000055eb1bb4bb7f in multifd_send_thread (opaque=0x55eb1e2d19f8) at migration/ram.c:1171 #7 0x000055eb1bfad628 in qemu_thread_start (args=0x55eb1e170450) at util/qemu-thread-posix.c:502 #8 0x00007f81a2b36df5 in start_thread () from /usr/lib64/libpthread.so.0 #9 0x00007f81a286048d in clone () from /usr/lib64/libc.so.6 To fix it up, let's destroy the mutex after all the other multifd threads had been terminated. Signed-off-by: Jiahui Cen <cenjiahui@huawei.com> Signed-off-by: Ying Fang <fangying1@huawei.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> (cherry picked from commit 9560a48ecc0c20d87bc458a6db77fba651605819) Signed-off-by: Danilo C. L. de Paula <ddepaula@redhat.com> --- migration/ram.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 860f781..6c55c5d 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -1052,6 +1052,10 @@ void multifd_save_cleanup(void) if (p->running) { qemu_thread_join(&p->thread); } + } + for (i = 0; i < migrate_multifd_channels(); i++) { + MultiFDSendParams *p = &multifd_send_state->params[i]; + socket_send_channel_destroy(p->c); p->c = NULL; qemu_mutex_destroy(&p->mutex); @@ -1335,6 +1339,10 @@ int multifd_load_cleanup(Error **errp) qemu_sem_post(&p->sem_sync); qemu_thread_join(&p->thread); } + } + for (i = 0; i < migrate_multifd_channels(); i++) { + MultiFDRecvParams *p = &multifd_recv_state->params[i]; + object_unref(OBJECT(p->c)); p->c = NULL; qemu_mutex_destroy(&p->mutex); -- 1.8.3.1