[ This is most likely a duplicate of PR21051. Filing it separately to prevent obfuscating original PR. ] Using gdb build from HEAD of today (2019-04-15): ... $ git show -s --pretty="%h %s" 8669f96f0d Automatic date update in version.in ... Terminal 1: ... $ ./gdb /usr/bin/sleep ... Terminal 2: ... $ ./gdb -p $(ps -u $USER | grep gdb | awk '{print $1}') ... Terminal 3: ... $ ./gdb -p $(ps -u $USER | grep gdb | awk '{print $1}' | tail -n 1) (gdb) continue Continuing. ... Terminal 2: ... (gdb) set follow-fork-mode child (gdb) continue Continuing. ... Terminal 1: ... (gdb) run Starting program: /usr/bin/sleep warning: Could not trace the inferior process. Error: warning: ptrace: Action is not allowed ... Terminal 2: ... [Attaching after Thread 0x7fd9945c9ac0 (LWP 15665) vfork to child process 15722] [New inferior 2 (process 15722)] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [Detaching vfork parent process 15665 after child exit] /data/gdb_versions/devel/src/gdb/nat/x86-linux-dregs.c:146: internal-error: void x86_linux_update_debug_registers(lwp_info*): Assertion `lwp_is_stopped (lwp)' failed. A problem internal to GDB has been detected, further debugging may prove unreliable. Quit this debugging session? (y or n) ... Terminal 3: ... ^C Thread 1 "gdb" received signal SIGINT, Interrupt. 0x00007f8c2181807b in poll () from /lib64/libc.so.6 (gdb) bt #0 0x00007f8c2181807b in poll () from /lib64/libc.so.6 #1 0x000000000061bebb in gdb_wait_for_event (block=1) at gdb/event-loop.c:770 #2 0x000000000061b2c2 in gdb_do_one_event () at gdb/event-loop.c:347 #3 0x00000000008caf14 in gdb_readline_wrapper ( prompt=0x3363230 "gdb/nat/x86-linux-dregs.c:146: internal-error: void x86_linux_update_debug_registers(lwp_info*): Assertion `lwp_is_stopped (lwp)' failed.\nA problem internal to GDB has bee"...) at gdb/top.c:1004 #4 0x00000000009278c1 in defaulted_query(const char *, char, typedef __va_list_tag __va_list_tag *) ( ctlstr=0xc66b68 "%s\nQuit this debugging session? ", defchar=0 '\000', args=0x7ffec3aca140) at gdb/utils.c:894 #5 0x0000000000927c79 in query (ctlstr=0xc66b68 "%s\nQuit this debugging session? ") at gdb/utils.c:986 #6 0x0000000000926a39 in internal_vproblem(internal_problem *, const char *, int, const char *, typedef __va_list_tag __va_list_tag *) (problem=0x11ac940 <internal_error_problem>, file=0xbf4fd8 "gdb/nat/x86-linux-dregs.c", line=146, fmt=0xbf4faa "%s: Assertion `%s' failed.", ap=0x7ffec3aca348) at gdb/utils.c:370 #7 0x0000000000926ccc in internal_verror ( file=0xbf4fd8 "gdb/nat/x86-linux-dregs.c", line=146, fmt=0xbf4faa "%s: Assertion `%s' failed.", ap=0x7ffec3aca348) at gdb/utils.c:436 #8 0x0000000000512f45 in internal_error ( file=0xbf4fd8 "gdb/nat/x86-linux-dregs.c", line=146, fmt=0xbf4faa "%s: Assertion `%s' failed.") at gdb/common/errors.c:55 #9 0x000000000076667c in x86_linux_update_debug_registers (lwp=0x3378ac0) at gdb/nat/x86-linux-dregs.c:146 #10 0x000000000076688d in x86_linux_prepare_to_resume (lwp=0x3378ac0) at gdb/nat/x86-linux.c:81 #11 0x000000000044c3f6 in x86_linux_nat_target::low_prepare_to_resume (this=0x11bd5b0 <the_amd64_linux_nat_target>, lwp=0x3378ac0) at gdb/x86-linux-nat.h:69 #12 0x00000000006f2661 in detach_one_lwp (lp=0x3378ac0, signo_p=0x0) at gdb/linux-nat.c:1406 #13 0x00000000006f2951 in detach_callback (lp=0x3378ac0) at gdb/linux-nat.c:1464 #14 0x00000000006fa7bc in gdb::function_view<int (lwp_info*)>::bind<int, lwp_info*>(int (*)(lwp_info*))::{lambda(gdb::fv_detail::erased_callable, lwp_info*)#1}::operator()(gdb::fv_detail::erased_callable, lwp_info*) const (__closure=0x0, ecall=..., args#0=0x3378ac0) at gdb/common/function-view.h:284 #15 0x00000000006fa7e3 in gdb::function_view<int (lwp_info*)>::bind<int, lwp_info*>(int (*)(lwp_info*))::{lambda(gdb::fv_detail::erased_callable, lwp_info*)#1}::_FUN(gdb::fv_detail::erased_callable, lwp_info*) () at gdb/common/function-view.h:278 #16 0x00000000006fa676 in gdb::function_view<int (lwp_info*)>::operator()(lwp_info*) const (this=0x7ffec3aca650, args#0=0x3378ac0) at gdb/common/function-view.h:247 #17 0x00000000006f1760 in iterate_over_lwps(ptid_t, gdb::function_view<int (lwp_info*)>) (filter=..., callback=...) at gdb/linux-nat.c:972 #18 0x00000000006f2a8e in linux_nat_target::detach (this=0x11bd5b0 <the_amd64_linux_nat_target>, inf=0x334c5e0, from_tty=0) at gdb/linux-nat.c:1484 #19 0x0000000000709a21 in thread_db_target::detach (this=0x11a1978 <the_thread_db_target>, inf=0x334c5e0, from_tty=0) at gdb/linux-thread-db.c:1364 #20 0x00000000008a9742 in target_detach (inf=0x334c5e0, from_tty=0) at gdb/target.c:2038 #21 0x00000000006b9c44 in handle_vfork_child_exec_or_exit (exec=0) at gdb/infrun.c:985 #22 0x00000000006c16b9 in handle_inferior_event (ecs=0x7ffec3acab80) at gdb/infrun.c:4830 #23 0x00000000006bf204 in fetch_inferior_event (client_data=0x0) at gdb/infrun.c:3748 #24 0x00000000006a6e8c in inferior_event_handler (event_type=INF_REG_EVENT, client_data=0x0) at gdb/inf-loop.c:43 #25 0x00000000006f9218 in handle_target_event (error=0, client_data=0x0) at gdb/linux-nat.c:4355 #26 0x000000000061be10 in handle_file_event (file_ptr=0x5393160, ready_mask=1) at gdb/event-loop.c:732 #27 0x000000000061c3b3 in gdb_wait_for_event (block=0) at gdb/event-loop.c:858 #28 0x000000000061b24e in gdb_do_one_event () at gdb/event-loop.c:322 #29 0x000000000061b2ec in start_event_loop () at gdb/event-loop.c:371 #30 0x000000000071df7f in captured_command_loop () at gdb/main.c:331 #31 0x000000000071f3e2 in captured_main (data=0x7ffec3acaef0) at gdb/main.c:1173 #32 0x000000000071f473 in gdb_main (args=0x7ffec3acaef0) at gdb/main.c:1188 #33 0x000000000040fcde in main (argc=3, argv=0x7ffec3acaff8) at gdb/gdb.c:32 ...
(In reply to Tom de Vries from comment #0) > [ This is most likely a duplicate of PR21051. There is one difference though: here we have exec == 0 in handle_vfork_child_exec_or_exit: ... #21 0x00000000006b9c44 in handle_vfork_child_exec_or_exit (exec=0) at gdb/infrun.c:985 ... while exec == 1 in both examples in PR21051.
Consider this more minimal test-case, calling vfork from a thread: ... $ cat test.c #include <stdio.h> #include <unistd.h> #include <pthread.h> static void * f (void *arg) { printf("thread created\n"); if (vfork () == 0) printf("vfork child\n"); else printf("vfork parent\n"); printf("thread exiting\n"); return NULL; } int main (void) { pthread_t tid; printf("thread create\n"); pthread_create (&tid, NULL, f, NULL); printf("thread joining\n"); pthread_join (tid, NULL); printf("thread joined\n"); return 0; } ... Compiled like so: ... $ gcc test.c -lpthread ... executes without problems: ... $ ./a.out thread create thread joining thread created vfork child thread exiting vfork parent thread exiting thread joined $ echo $? 0 ... when running with gdb in follow-fork-mode child mode, we run into the assert: ... $ gdb -batch ./a.out -ex "set follow-fork-mode child" -ex "run" [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". thread create [New Thread 0x7ffff77fe700 (LWP 7557)] thread created thread joining [Attaching after Thread 0x7ffff77fe700 (LWP 7557) vfork to child process 7558] [New inferior 2 (process 7558)] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". vfork child thread exiting [Detaching vfork parent process 7553 after child exit] vfork parent thread exiting /data/gdb_versions/devel/src/gdb/nat/x86-linux-dregs.c:146: internal-error: void x86_linux_update_debug_registers(lwp_info*): Assertion `lwp_is_stopped (lwp)' failed. A problem internal to GDB has been detected, further debugging may prove unreliable. Quit this debugging session? (y or n) [answered Y; input not from terminal] This is a bug, please report it. For instructions, see: <http://www.gnu.org/software/gdb/bugs/>. /data/gdb_versions/devel/src/gdb/nat/x86-linux-dregs.c:146: internal-error: void x86_linux_update_debug_registers(lwp_info*): Assertion `lwp_is_stopped (lwp)' failed. A problem internal to GDB has been detected, further debugging may prove unreliable. Create a core file of GDB? (y or n) [answered Y; input not from terminal] Afgebroken (geheugendump gemaakt) ...
Created attachment 11742 [details] 0001-gdb-testsuite-Add-vfork-follow-child.exp-test-case.patch [gdb/testsuite] Add vfork-follow-child.exp test-case PR gdb/24454 Test-case for PR gdb/24454. Fails like this: ... Running src/gdb/testsuite/gdb.threads/vfork-follow-child.exp ... FAIL: gdb.threads/vfork-follow-child.exp: continue (GDB internal error) ERROR: Could not resync from internal error (resync count exceeded) === gdb Summary === nr of expected passes 2 nr of unexpected failures 1 ... --- gdb/testsuite/gdb.threads/vfork-follow-child.c | 19 +++++++++++++++++++ gdb/testsuite/gdb.threads/vfork-follow-child.exp | 21 +++++++++++++++++++++ 2 files changed, 40 insertions(+)
Tentative patch: ... diff --git a/gdb/infrun.c b/gdb/infrun.c index 37713b24fe..80978d5198 100644 --- a/gdb/infrun.c +++ b/gdb/infrun.c @@ -982,7 +982,14 @@ handle_vfork_child_exec_or_exit (int exec) } } - target_detach (inf->vfork_parent, 0); + /* Now that the vfork child has terminated, make sure during detach + that we no longer consider the vfork parent to be a vfork parent, + but just a regular process that we're detaching from. */ + struct inferior *to_detach = inf->vfork_parent; + inf->vfork_parent->vfork_child = NULL; + inf->vfork_parent = NULL; + + target_detach (to_detach, 0); /* Put it back. */ inf->pspace = pspace; ...
(In reply to Tom de Vries from comment #4) > Tentative patch: Submitted: https://sourceware.org/ml/gdb-patches/2019-04/msg00271.html
The master branch has been updated by Pedro Alves <palves@sourceware.org>: https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=b73715df01e6e9b3de5a49cd7bf4170deef48461 commit b73715df01e6e9b3de5a49cd7bf4170deef48461 Author: Tom de Vries <tdevries@suse.de> Date: Thu Apr 18 17:05:43 2019 +0100 [gdb] Handle vfork in thread with follow-fork-mode child When debugging any of the testcases added by this commit, which do a vfork in a thread with "set follow-fork-mode child" + "set detach-on-fork on", we run into this assertion: ... src/gdb/nat/x86-linux-dregs.c:146: internal-error: \ void x86_linux_update_debug_registers(lwp_info*): \ Assertion `lwp_is_stopped (lwp)' failed. ... The assert is caused by the following: the vfork-child exit or exec event is handled by handle_vfork_child_exec_or_exit, which calls target_detach to detach from the vfork parent. During target_detach we call linux_nat_target::detach, which: #1 - stops all the threads #2 - waits for all the threads to be stopped #3 - detaches all the threads However, during the second step we run into this code in stop_wait_callback: ... /* If this is a vfork parent, bail out, it is not going to report any SIGSTOP until the vfork is done with. */ if (inf->vfork_child != NULL) return 0; ... and we don't wait for the threads to be stopped, which results in this assert in x86_linux_update_debug_registers triggering during the third step: ... gdb_assert (lwp_is_stopped (lwp)); ... The fix is to reset the vfork parent's vfork_child field before calling target_detach in handle_vfork_child_exec_or_exit. There's already similar code for the other paths handled by handle_vfork_child_exec_or_exit, so this commit refactors the code a bit so that all paths share the same code. The new tests cover both a vfork child exiting, and a vfork child execing, since both cases would trigger the assertion. The new testcases also exercise following the vfork children with "set detach-on-fork off", since it doesn't seem to be tested anywhere. Tested on x86_64-linux, using native and native-gdbserver. gdb/ChangeLog: 2019-04-18 Tom de Vries <tdevries@suse.de> Pedro Alves <palves@redhat.com> PR gdb/24454 * infrun.c (handle_vfork_child_exec_or_exit): Reset vfork parent's vfork_child field before calling target_detach. gdb/testsuite/ChangeLog: 2019-04-18 Tom de Vries <tdevries@suse.de> Pedro Alves <palves@redhat.com> PR gdb/24454 * gdb.threads/vfork-follow-child-exec.c: New file. * gdb.threads/vfork-follow-child-exec.exp: New file. * gdb.threads/vfork-follow-child-exit.c: New file. * gdb.threads/vfork-follow-child-exit.exp: New file.
Patch with test-cases committed, marking resolved-fixed.
*** Bug 21051 has been marked as a duplicate of this bug. ***
*** Bug 24553 has been marked as a duplicate of this bug. ***
proposed 8.3 backport: https://sourceware.org/ml/gdb-patches/2019-07/
The gdb-8.3-branch branch has been updated by Tom de Vries <vries@sourceware.org>: https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;h=ee479c89ed9ecd446eccbbdb0b8786c6526f8906 commit ee479c89ed9ecd446eccbbdb0b8786c6526f8906 Author: Tom de Vries <tdevries@suse.de> Date: Fri Aug 16 00:31:48 2019 +0200 [gdb] Handle vfork in thread with follow-fork-mode child [ Backport of master commit b73715df01. ] When debugging any of the testcases added by this commit, which do a vfork in a thread with "set follow-fork-mode child" + "set detach-on-fork on", we run into this assertion: ... src/gdb/nat/x86-linux-dregs.c:146: internal-error: \ void x86_linux_update_debug_registers(lwp_info*): \ Assertion `lwp_is_stopped (lwp)' failed. ... The assert is caused by the following: the vfork-child exit or exec event is handled by handle_vfork_child_exec_or_exit, which calls target_detach to detach from the vfork parent. During target_detach we call linux_nat_target::detach, which: However, during the second step we run into this code in stop_wait_callback: ... /* If this is a vfork parent, bail out, it is not going to report any SIGSTOP until the vfork is done with. */ if (inf->vfork_child != NULL) return 0; ... and we don't wait for the threads to be stopped, which results in this assert in x86_linux_update_debug_registers triggering during the third step: ... gdb_assert (lwp_is_stopped (lwp)); ... The fix is to reset the vfork parent's vfork_child field before calling target_detach in handle_vfork_child_exec_or_exit. There's already similar code for the other paths handled by handle_vfork_child_exec_or_exit, so this commit refactors the code a bit so that all paths share the same code. The new tests cover both a vfork child exiting, and a vfork child execing, since both cases would trigger the assertion. The new testcases also exercise following the vfork children with "set detach-on-fork off", since it doesn't seem to be tested anywhere. Tested on x86_64-linux, using native and native-gdbserver. gdb/ChangeLog: 2019-04-18 Tom de Vries <tdevries@suse.de> Pedro Alves <palves@redhat.com> PR gdb/24454 * infrun.c (handle_vfork_child_exec_or_exit): Reset vfork parent's vfork_child field before calling target_detach. gdb/testsuite/ChangeLog: 2019-04-18 Tom de Vries <tdevries@suse.de> Pedro Alves <palves@redhat.com> PR gdb/24454 * gdb.threads/vfork-follow-child-exec.c: New file. * gdb.threads/vfork-follow-child-exec.exp: New file. * gdb.threads/vfork-follow-child-exit.c: New file. * gdb.threads/vfork-follow-child-exit.exp: New file.
changing the target milestone because the patch was backported there as well.
#include <locale.h> #include <stdio.h> #include <wchar.h> int main(void) { setlocale(LC_ALL, ""); fgetwc(stdin); return 0; } https://komiya-dental.com/ --- $ gcc -o poc poc.c $ python -c 'print 13*"\t"' | LC_CTYPE=en_US.UTF-8 ./poc Segmentation fault $ python -c 'print 13*"\t"' | LC_CTYPE=POSIX ./poc $ _ It means that I have to enter around 13 tabulator characters to trigger the issue, but it won't hurt to add a few more. I was able to reproduce this on other distributions with glibc 2.24, so I don't think that it's specific to one of them. Also, this issue only happens with an LC_CTYPE of an UTF-8 locale. I have tested en_US and de_DE, which both trigger this issue. With POSIX or C, the segmentation fault is not triggered. I hope this helps you to track down this bug, as I was unable to figure out the flush mechanisms in glibc in a reasonable time. :) The stack trace on my system with glibc 2.24 looks like this: http://www.iu-bloomington.com/ (gdb) bt #0 __GI__IO_wfile_sync (fp=0xb77295a0 <_IO_2_1_stdin_>) at wfileops.c:534 #1 0xb75e2bc6 in _IO_default_setbuf (fp=0xb77295a0 <_IO_2_1_stdin_>, p=0x0, len=0) at genops.c:523 #2 0xb75df2e2 in _IO_new_file_setbuf (fp=0xb77295a0 <_IO_2_1_stdin_>, p=0x0, len=0) at fileops.c:459 https://www.webb-dev.co.uk/ #3 0xb75e3516 in _IO_unbuffer_all () at genops.c:921 #4 _IO_cleanup () at genops.c:966 #5 0xb75a5632 in __run_exit_handlers (status=0, listp=0xb77293dc <__exit_funcs>, run_list_atexit=true, run_dtors=true) at exit.c:96 #6 0xb75a56f1 in __GI_exit (status=0) at exit.c:105 #7 0xb758f1b2 in __libc_start_main (main=0x804846b <main>, argc=1, argv=0xbfef4004, init=0x80484b0 <__libc_csu_init>, fini=0x8048510 <__libc_csu_fini>, rtld_fini=0xb774d7a0 <_dl_fini>, stack_end=0xbfef3ffc) at ../csu/libc-start.c:323 https://waytowhatsnext.com/ #include <locale.h> #include <stdio.h> #include <wchar.h> int main(void) http://www.acpirateradio.co.uk/ { setlocale(LC_ALL, ""); fgetwc(stdin); return 0; } --- $ gcc -o poc poc.c $ python -c 'print 13*"\t"' | LC_CTYPE=en_US.UTF-8 ./poc Segmentation fault $ python -c 'print 13*"\t"' | LC_CTYPE=POSIX ./poc $ _ http://www.logoarts.co.uk/ It means that I have to enter around 13 tabulator characters to trigger the issue, but it won't hurt to add a few more. I was able to reproduce this on other distributions with glibc 2.24, so I don't think that it's specific to one of them. Also, this issue only happens with an LC_CTYPE of an UTF-8 locale. I have tested en_US and de_DE, which both trigger this issue. With POSIX or C, the http://www.slipstone.co.uk/ segmentation fault is not triggered. I hope this helps you to track down this bug, as I was unable to figure out the flush mechanisms in glibc in a reasonable time. :) http://embermanchester.uk/ The stack trace on my system with glibc 2.24 looks like this: http://connstr.net/ (gdb) bt #0 __GI__IO_wfile_sync (fp=0xb77295a0 <_IO_2_1_stdin_>) at wfileops.c:534 http://joerg.li/ #1 0xb75e2bc6 in _IO_default_setbuf (fp=0xb77295a0 <_IO_2_1_stdin_>, p=0x0, len=0) at genops.c:523 #2 0xb75df2e2 in _IO_new_file_setbuf (fp=0xb77295a0 <_IO_2_1_stdin_>, p=0x0, http://www.jopspeech.com/ len=0) at fileops.c:459 #3 0xb75e3516 in _IO_unbuffer_all () at genops.c:921 #4 _IO_cleanup () at genops.c:966 http://www.wearelondonmade.com/ #5 0xb75a5632 in __run_exit_handlers (status=0, listp=0xb77293dc <__exit_funcs>, run_list_atexit=true, run_dtors=true) at exit.c:96 #6 0xb75a56f1 in __GI_exit (status=0) at exit.c:105 http://www.compilatori.com/ #7 0xb758f1b2 in __libc_start_main (main=0x804846b <main>, argc=1, argv=0xbfef4004, init=0x80484b0 <__libc_csu_init>, fini=0x8048510 <__libc_csu_fini>, http://www-look-4.com/ rtld_fini=0xb774d7a0 <_dl_fini>, stack_end=0xbfef3ffc) at ../csu/libc-start.c:323
https://www.ремонты-квартир.com/ https://www.дизайн-квартиры.com/ https://www.о-ремонте.com/ https://www.о-заборах.com/ https://www.bsegypt.com/ https://www.buyingrealty.net/ https://www.khersonnews.com/ https://www.kontrolstroy.info/ https://www.sama-mama.com/ https://www.secretovnet.org/ https://www.teleriko.com/ https://www.us-best-store.com/ https://www.віктор.com/ https://www.accord-hotel.ru/ https://releazer.ru/ https://www.a-n-e-k-d-o-t.ru/ https://www.adhan.ru/ http://www.al-aures.ru/ https://www.apriori-design.ru/ http://artdoski.ru/ https://www.bombusmod.net.ru/ https://www.canadianahealthandcaremallreviews.ru/ https://www.celestiaproject.ru/ https://www.cryptogu.ru/ https://www.downloadskypefree.ru/ https://www.encyclopedia-flowers.ru/ https://www.factura.net.ru/ http://freewizards.ru/ http://futurefactory.ru/ https://glina-med.ru/ http://google-dmoz.ru/ http://iix.su/ https://www.imperia51.ru/ https://www.info-tehnologii.ru/ https://www.kvartira-v-bolgarii.ru/ https://ljubi-i-pozdravljaj.ru/ https://www.majesticarticles.ru/ https://www.onlinecredit247.ru/ https://www.orfey.net.ru/ https://www.pgpk.net.ru/ https://www.rainbow.net.ru/ http://www.rainbowbaby.ru/ http://www.respublika-okon.ru/ https://ribku-lovim.ru/ http://rusorchestra.ru/ http://shmoscow.ru/ https://www.skifspb.ru/ https://www.spare.net.ru/ https://www.stranainform.ru/ https://www.taxi-smile.ru/ https://www.tkanishik.ru/ http://www.tremulous.net.ru/ https://trust-women.ru/ http://uralbel.ru/ https://www.yar-art-union.ru/ https://www.xn----7sbcngq4awkg0k.xn--p1ai/ https://www.xn----7sbbmgbytlh3a0ll.xn--p1ai/ https://www.xn--35-mlcuxidl.xn--p1ai/ https://www.xn--f1addf1alkk1d.xn--p1ai/ https://www.history-of-great-discoveries.com/ https://www.it-business-trends.com https://www.interesting-history-of-art.com https://www.interesting-news-about-cars.com https://www.architecture-and-design-news.com https://history-of-great-discoveries.blogspot.com/ https://it-business-trends.blogspot.com/ https://interesting-history-of-art.blogspot.com/ https://interesting-news-about-cars.blogspot.com/ https://architecture-and-design-news.blogspot.com/ https://www.secretovnet.org/archives/18806 https://www.secretovnet.org/archives/17685 https://www.secretovnet.org/archives/17683 https://www.secretovnet.org/archives / 17681 https://www.secretovnet.org/archives/13740 https://www.secretovnet.org/archives/13737 https://www.secretovnet.org/archives/13734 https://www.secretovnet.org / archives / 13732 https://www.secretovnet.org/archives/13729 https://www.secretovnet.org/archives/17679 https://www.secretovnet.org/archives/17677 https://www.secretovnet .org / archives / 17675 https://www.secretovnet.org/archives/17670 https://www.secretovnet.org/archives/17667 https://www.secretovnet.org/archives/18686 https://www.secretovnet.org/archives/18684 https://www.secretovnet.org/archives/18682 https://www.secretovnet.org/archives/17665 https://www.secretovnet.org/archives / 17663 https://www.secretovnet.org/archives/17661 https://www.secretovnet.org/archives/17659 https://www.secretovnet.org/archives/17657 https://www.secretovnet.org / archives / 13723 https://www.secretovnet.org/archives/13717 https://www.secretovnet.org/archives/13714 https://www.secretovnet.org/archives/13711 https://www.secretovnet .org / archives / 13708 https://www.secretovnet.org/archives/17655 https://www.secretovnet.org/archives/13702 https://www.secretovnet.org/archives/17647 https://www.secretovnet.org/archives/17645
http://www.ремонты-квартир.com/ http://www.дизайн-квартиры.com/ http://www.о-ремонте.com/ http://www.о-заборах.com/ http://www.bsegypt.com/ http://www.buyingrealty.net/ http://www.khersonnews.com/ http://www.kontrolstroy.info/ http://www.sama-mama.com/ http://www.secretovnet.org/ http://www.teleriko.com/ http://www.us-best-store.com/ http://www.віктор.com/ http://www.accord-hotel.ru/ http://releazer.ru/ http://www.a-n-e-k-d-o-t.ru/ http://www.adhan.ru/ https://www.al-aures.ru/ http://www.apriori-design.ru/ https://artdoski.ru/ http://www.bombusmod.net.ru/ http://www.canadianahealthandcaremallreviews.ru/ http://www.celestiaproject.ru/ http://www.cryptogu.ru/ http://www.downloadskypefree.ru/ http://www.encyclopedia-flowers.ru/ http://www.factura.net.ru/ https://freewizards.ru/ https://futurefactory.ru/ http://glina-med.ru/ https://google-dmoz.ru/ https://iix.su/ http://www.imperia51.ru/ http://www.info-tehnologii.ru/ http://www.kvartira-v-bolgarii.ru/ http://ljubi-i-pozdravljaj.ru/ http://www.majesticarticles.ru/ http://www.onlinecredit247.ru/ http://www.orfey.net.ru/ http://www.pgpk.net.ru/ http://www.rainbow.net.ru/ https://www.rainbowbaby.ru/ https://www.respublika-okon.ru/ http://ribku-lovim.ru/ https://rusorchestra.ru/ https://shmoscow.ru/ http://www.skifspb.ru/ http://www.spare.net.ru/ http://www.stranainform.ru/ http://www.taxi-smile.ru/ http://www.tkanishik.ru/ https://www.tremulous.net.ru/ http://trust-women.ru/ https://uralbel.ru/ http://www.yar-art-union.ru/ http://www.xn----7sbcngq4awkg0k.xn--p1ai/ http://www.xn----7sbbmgbytlh3a0ll.xn--p1ai/ http://www.xn--35-mlcuxidl.xn--p1ai/ http://www.xn--f1addf1alkk1d.xn--p1ai/ http://www.history-of-great-discoveries.com/ http://www.it-business-trends.com http://www.interesting-history-of-art.com http://www.interesting-news-about-cars.com http://www.architecture-and-design-news.com https://ремонты-квартир.com/ https://дизайн-квартиры.com/ https://о-ремонте.com/ https://о-заборах.com/ https://bsegypt.com/ https://buyingrealty.net/ https://khersonnews.com/ https://kontrolstroy.info/ https://sama-mama.com/ https://secretovnet.org/ https://teleriko.com/ https://us-best-store.com/ https://віктор.com/ https://accord-hotel.ru/ https://www.releazer.ru/ https://a-n-e-k-d-o-t.ru/ https://adhan.ru/ http://al-aures.ru/ https://apriori-design.ru/ http://www.artdoski.ru/ https://bombusmod.net.ru/ https://canadianahealthandcaremallreviews.ru/ https://celestiaproject.ru/ https://cryptogu.ru/ https://downloadskypefree.ru/ https://encyclopedia-flowers.ru/ https://factura.net.ru/ http://www.freewizards.ru/ http://www.futurefactory.ru/ https://www.glina-med.ru/ http://www.google-dmoz.ru/ http://www.iix.su/ https://imperia51.ru/ https://info-tehnologii.ru/ https://kvartira-v-bolgarii.ru/ https://www.ljubi-i-pozdravljaj.ru/ https://majesticarticles.ru/ https://onlinecredit247.ru/ https://orfey.net.ru/ https://pgpk.net.ru/ https://rainbow.net.ru/ http://rainbowbaby.ru/ http://respublika-okon.ru/ https://www.ribku-lovim.ru/ http://www.rusorchestra.ru/ http://www.shmoscow.ru/ https://skifspb.ru/ https://spare.net.ru/ https://stranainform.ru/ https://taxi-smile.ru/ https://tkanishik.ru/ http://tremulous.net.ru/ https://www.trust-women.ru/ http://www.uralbel.ru/ https://yar-art-union.ru/ https://xn----7sbcngq4awkg0k.xn--p1ai/ https://xn----7sbbmgbytlh3a0ll.xn--p1ai/ https://xn--35-mlcuxidl.xn--p1ai/ https://xn--f1addf1alkk1d.xn--p1ai/ https://history-of-great-discoveries.com/ https://it-business-trends.com https://interesting-history-of-art.com https://interesting-news-about-cars.com https://architecture-and-design-news.com