John McEleney / mailutils

New ioctls MU_IOCTL_GET_TRANSPORT_BUFFER and
MU_IOCTL_SET_TRANSPORT_BUFFER return and modify bufferization
mode in the lowest level transport stream.  Both server and
client programs use this to switch to full buffering before
sending large amounts of data. This has a particular impact
on the output speed and CPU usage when TLS is in use.

* include/mailutils/stream.h (MU_IOCTL_GET_TRANSPORT_BUFFER)
(MU_IOCTL_SET_TRANSPORT_BUFFER): New ioctls.
(MU_TRANSPORT_INPUT, MU_TRANSPORT_OUTPUT): New constants.
(MU_TRANSPORT_VALID_TYPE): New macro.
(mu_buffer_query): New struct.
(mu_stream_get_buffer): New proto.

* libmu_auth/tls.c (_tls_io_ioctl): Return ENOSYS
if op is not supported.
(_tls_ioctl): Support MU_IOCTL_[GS]ET_TRANSPORT_BUFFER.
* mailbox/file_stream.c (fd_ioctl): Support MU_IOCTL_[GS]ET_TRANSPORT_BUFFER.
Return ENOSYS if op is not supported.
* mailbox/filter_iconv.c (_icvt_ioctl): Likewise.
* mailbox/iostream.c (_iostream_ctl): Likewise.
* mailbox/mapfile_stream.c (_mapfile_ioctl): Likewise.
* mailbox/memory_stream.c (_memory_ioctl): Likewise.
* mailbox/rdcache_stream.c (rdcache_ioctl): Likewise.
* mailbox/xscript-stream.c (_xscript_ctl): Likewise.
* mailbox/prog_stream.c (_prog_ioctl): Return ENOSYS if op is not
supported.
* mailbox/tcp.c (_tcp_ioctl): Likewise.

* mailbox/stream.c (mu_stream_default_buffer_size): New global.
(mu_stream_seek): Do not call seek method if the requested offset
is the same as the current one.
(mu_stream_set_buffer): Size==0 means use the default value.
(mu_stream_get_buffer): New function.
(mu_stream_truncate): Flush the buffer before truncating.

* mailbox/mime.c (mime_reset_state): New function.
(_mime_body_stream_seek): Rewrite using mime_reset_state.
(_mime_body_stream_ioctl): Return ENOSYS if op is not
supported.
(create_mime_body_stream): Reset mime state (_mime_body_stream_seek
may not be called due to the optimization in mu_stream_seek.

* mailbox/header.c (header_parse): Exit correctly if the buffer is
not terminated with a \n.

* libproto/mbox/mbox.c (mbox_open): Enforce full buffering on the
mailbox stream.
* libproto/pop/mbox.c (pop_header_fill): Minor fix.
* libproto/pop/pop3_stream.c (_POP3F_DONE)
(_POP3F_CHBUF): New constants.
(mu_pop3_stream)<done>: Remove. Replaced with flags.
<oldbuf>: New member.
(_pop3_event_cb): If _POP3F_CHBUF flag is set, restore
the initial buffering mode on the transport stream.
(mu_pop3_filter_create): Save away current buffering mode
and enforce full buffering while transferring bulk data.

* mail/from.c (hdr_from): Check return value from mu_message_get_header.
* mail/mail.c (main): Allow for MAILRC having empty value.
Always close the mailbox before exiting.

* pop3d/pop3d.c (pop3d_cfg_param): New statement output-buffer-size.
* pop3d/pop3d.h (pop3d_output_bufsize): New global.
* pop3d/retr.c (pop3d_send_payload): New function.
(pop3d_retr): Rewrite using pop3d_send_payload. Save away
current buffering mode and enforce full buffering while
transferring bulk data.
* pop3d/top.c (pop3d_top_: Likewise.

Sergey Poznyakoff authored 2010-09-09 23:18:40 +0300

* mailbox/url.c (ACCESSOR(is_same,field)): Fix coredump if one
of the accessors returned MU_ERR_NOENT.

Sergey Poznyakoff authored 2010-09-09 18:19:28 +0300

Sergey Poznyakoff authored 2010-09-09 11:31:41 +0300

Sergey Poznyakoff authored 2010-09-08 16:11:38 +0300

* include/mailutils/stream.h (MU_IOCTL_LEVEL): New ioctl op.
(XSCRIPT_NORMAL, XSCRIPT_SECURE, XSCRIPT_PAYLOAD): New constants.
* include/mailutils/sys/xscript-stream.h (_mu_xscript_stream)
<level>: New member.
* mailbox/xscript-stream.c (TRANS_DISABLED): New flag.
(print_transcript): Amount of output varies depending on the
current output level.  For secure data, try to recognize passwords
and to replace them with *** on output.
(_xscript_ctl): Support MU_IOCTL_LEVEL.
* pop3d/extra.c (set_xscript_level): New function.
* pop3d/pop3d.h (set_xscript_level): New proto.
* pop3d/retr.c (pop3d_retr): Set XSCRIPT_PAYLOAD level before
sending actual data and reset it to XSCRIPT_NORMAL afterwards.
* pop3d/top.c (pop3d_top): Likewise.
* pop3d/user.c: Set XSCRIPT_SECURE level while expecting the
PASS command.

* imap4d/fetch.c (imap4d_fetch): Run imap4d_fetch0 in XSCRIPT_PAYLOAD
level.
* imap4d/uid.c (imap4d_uid): Likewise.
* imap4d/imap4d.c (imap4d_mainloop): Unless started in preauth mode,
select XSCRIPT_SECURE mode until authentication has been passed.
* imap4d/imap4d.h (set_xscript_level): New proto.
* imap4d/io.c (io_format_completion_response): Switch to XSCRIPT_NORMAL
level when changing to the authenticated state.
(imap4d_readline): Read literals in XSCRIPT_PAYLOAD level.
* imap4d/util.c (set_xscript_level): New function.

* include/mailutils/pop3.h (mu_pop3_trace_mask): New prototype.
(MU_POP3_XSCRIPT_MASK): New macro.
(_mu_pop3_xscript_level): New proto.
* libproto/pop/pop3_pass.c (mu_pop3_pass): Set XSCRIPT_SECURE
while sending the password.
* libproto/pop/pop3_retr.c (mu_pop3_retr): Set XSCRIPT_PAYLOAD before
going to MU_POP3_RETR_RX state.
* libproto/pop/pop3_stream.c (_pop3_event_cb): Set XSCRIPT_NORMAL.
* libproto/pop/pop3_top.c (mu_pop3_top): Set XSCRIPT_PAYLOAD before
going to MU_POP3_TOP_RX state.
* libproto/pop/pop3_trace.c (mu_pop3_trace_mask)
(_mu_pop3_xscript_level): New functions.
* libproto/pop/mbox.c (pop_open): Set trace masks depending on the
trace6 and trace7 debug levels.

* examples/pop3client.c (com_verbose): Allow to mask/unmask transcript
levels.

Sergey Poznyakoff authored 2010-09-08 16:11:38 +0300

* mailbox/msgscan.c: New file.
* mailbox/Makefile.am (libmailutils_la_SOURCES): Add it.

* include/mailutils/body.h (mu_body_set_get_stream): New prototype.
* include/mailutils/message.h (MU_SCAN_SEEK, MU_SCAN_SIZE): New
defines.
(mu_message_scan): New structure.
(mu_stream_scan_message): New prototype.
(mu_message_set_get_stream): New prototype.
* include/mailutils/stream.h (mu_stream_copy): Change signature: takes
4 arguments now.
* include/mailutils/sys/body.h (_mu_body) <_get_stream>: New method.
* include/mailutils/sys/message.h (_mu_message) <_get_stream>: New method.
* mailbox/body.c (_body_get_stream): Call _get_stream, if provided.
* mailbox/message.c (_message_get_stream): Call _get_stream, if provided.

* mailbox/stream.c (_stream_flush_buffer): Avoid infinite recursion:
call stream->seek directly.
* mailbox/streamcpy.c (mu_stream_copy): Return the number of bytes
actually copied in the fourth argument. All uses updated.
* mailbox/streamref.c (streamref_return): Do not propagate internal
flags.
(_streamref_readdelim): Ensure there is enough buffer space for the
mu_stream_readdelim call.

* libproto/pop/mbox.c: Finish client implementation.

* mail/print.c (mail_print_msg): Close pager before returning on
error.

Sergey Poznyakoff authored 2010-09-08 16:11:38 +0300

* include/mailutils/sys/stdio_stream.h: Remove.
* include/mailutils/sys/socket_stream.h: Remove.
* include/mailutils/sys/rdcache_stream.h: New file.
* include/mailutils/sys/Makefile.am: Update.
* mailbox/rdcache_stream.c: New file.
* mailbox/Makefile.am: Update.

* examples/mucat.c: New example.
* examples/musocio.c: New example.
* examples/Makefile.am (noinst_PROGRAMS): Build new examples.

* include/mailutils/stream.h (mu_fd_stream_create): New proto.
(mu_rdcache_stream_create): New proto.
* include/mailutils/sys/file_stream.h (_mu_file_stream_create): Change
prototype.
* mailbox/file_stream.c (fd_open): Raise the MU_STREAM_AUTOCLOSE bit.
(fd_ioctl): Support MU_IOCTL_SET_TRANSPORT.
(_mu_file_stream_create): Change signature. All uses updated.
Allocate a copy of the filename argument, unless it is NULL.
(mu_fd_stream_create): New function.

* mailbox/socket_stream.c: Rewrite using file_stream directly.
* mailbox/stdio_stream.c: Rewrite. Use rdcache_stream if
the seek capability is required on an input stream.
* mailbox/streamcpy.c (mu_stream_copy): Handle eventual
EACCES return from mu_stream_seek as equivalent to ENOSYS.

Sergey Poznyakoff authored 2010-09-08 16:11:37 +0300

Sergey Poznyakoff authored 2010-09-08 16:11:37 +0300

* examples/pop3client.c (com_capa): Call mu_pop3_capa_test.
(com_stat): Count is size_t.

* include/mailutils/opool.h (mu_opool_copy): New proto.
* mailbox/opool.c (mu_opool_copy): New function.
* mailbox/xscript-stream.c (_xscript_ctl)
<MU_IOCTL_SWAP_STREAM>: Avoid coredumping if sp->transport
is NULL.

* include/mailutils/pop3.h (pop3_capa_test): Rename to
mu_pop3_capa_test.
(mu_pop3_stat): Third argument is a pointer to mu_off_t.
* libproto/pop/pop3_capatst.c (pop3_capa_test): Rename to
mu_pop3_capa_test.
* libproto/pop/pop3_stat.c (mu_pop3_stat): Third argument is
a pointer to mu_off_t.

* libproto/pop/Makefile.am (libmu_pop_la_SOURCES): Put back
folder.c, url.c and mbox.c.
* libproto/pop/mbox.c: Begin rewriting.

Sergey Poznyakoff authored 2010-09-08 16:11:37 +0300

* include/mailutils/sys/stream.h (_MU_STR_EVENT_SET)
(_MU_STR_EVENT_CLR): New defines.
(_mu_stream) <event_cb, event_mask>: New members.
* mailbox/stream.c (_stream_setflag, _stream_clrflag): New static
functions.
All functions use these instead of setting/clearing flags directly.
(_mu_stream_cleareof, _mu_stream_seteof): New extern functions.
(_stream_cleareof): Remove define, use _mu_stream_cleareof instead.
(_stream_fill_buffer): Set EOF marker when end of stream is reached.

* mailbox/fltstream.c (filter_read): Call _mu_stream_seteof to set
EOF flag.

* include/mailutils/pop3.h: Get rid of the superfluous "extern" in
front of function prototypes.
Add new prototypes.
Remove extra whitespace.

* libproto/pop/pop3_capatst.c: New file.
* libproto/pop/pop3_list_cmd.c: New file.
* libproto/pop/pop3_listas.c: New file.
* libproto/pop/pop3_rdlist.c: New file.
* libproto/pop/pop3_uidl_cmd.c: New file.
* libproto/pop/pop3_uidlas.c: New file.
* libproto/pop/Makefile.am: Add new files.
* libproto/pop/pop3_capa.c (_mu_pop3_fill_list): Remove.
Use mu_pop3_read_list instead.
(capa_comp): New comparator for capa lists.
* libproto/pop/pop3_list.c (mu_pop3_list): Fix format specifier.
* libproto/pop/pop3_lista.c (mu_pop3_list_all): Rewrite.
* libproto/pop/pop3_retr.c (mu_pop3_retr) <MU_POP3_RETR_RX>: do not
reset state, this is done by the EOF event callback.
* libproto/pop/pop3_top.c (mu_pop3_top) <MU_POP3_TOP_RX>: Likewise.
* libproto/pop/pop3_stream.c (pop3_decode_state): New state pds_char.
Change semantics of pds_init.
(newstate, _pop3_decoder): Handle .\r\n in the initial state.
(_pop3_event_cb): New event callback.
(mu_pop3_filter_create): Set event callback on the filter stream.
* libproto/pop/pop3_uidla.c (mu_pop3_uidl_all): Rewrite.

* examples/Makefile.am (pop3client_CPPFLAGS): Add MU_APP_COMMON_INCLUDES.
* examples/pop3client.c: Rewrite command parser.

Sergey Poznyakoff authored 2010-09-08 16:11:37 +0300

* libproto/pop/pop3_create.c (_mu_pop3_init): Fix return value.
* libproto/pop/pop3_destroy.c: Add missing includes.
* libproto/pop/pop3_stream.c (_pop3_decoder): Remove the final
.\r\n altogether.
* mailbox/fltstream.c (filter_read): Remove the hack for supporting
buggy filters.

Sergey Poznyakoff authored 2010-09-08 16:11:37 +0300

* include/mailutils/tls.h (mu_tls_readline_fn)
(mu_tls_writeline_fn, mu_tls_stream_ctl_fn): Remove typedefs.
(mu_tls_begin): Remove prototype.
* libmu_auth/tls.c (mu_tls_begin): Remove function.
* libproto/mailer/smtp.c: Revamp STARTTLS support.

Sergey Poznyakoff authored 2010-09-08 16:11:36 +0300

* include/mailutils/filter.h (mu_filter_io) <eof>: New member.
* mailbox/fltstream.c (init_iobuf): Initialize eof to 0.
(filter_read): Break the loop if the decoder has set eof.
* mailbox/xscript-stream.c (_xscript_ctl) <MU_IOCTL_SWAP_STREAM>: Handle
a special case when the transport does not support stream swapping.

* include/mailutils/pop3.h (mu_pop3_set_debug): Remove.
(MU_POP3_TRACE_CLR, MU_POP3_TRACE_SET, MU_POP3_TRACE_QRY): New macros.
(mu_pop3_trace): New proto.
(mu_pop3_readline): Remove.
(mu_pop3_getline): New proto.
(mu_pop3_capa): Change signature.
* include/mailutils/sys/pop3.h (mu_pop3_state): Remove the *_ACK states.
(mu_pop3_work_buf): Remove.
(MU_POP3_ACK, MU_POP3_TRACE): New defines.
(_mu_pop3): Rewrite the structure.
(mu_pop3_debug_cmd, mu_pop3_debug_ack): Remove functions.
(MU_POP3_FISSET, MU_POP3_FSET, MU_POP3_FCLR): New macros.
(_mu_pop3_trace_enable, _mu_pop3_trace_disable)
(_mu_pop3_init): New protos.

* include/mailutils/tls.h (mu_tls_stream_ctl_fn)
(mu_tls_writeline_fn): Change typedefs.
* libmu_auth/tls.c (mu_tls_begin): Update function calls
accordingly.

* libproto/pop/pop3_debug.c: Remove.
* libproto/pop/pop3_trace.c: New function.
* libproto/pop/Makefile.am (libmu_pop_la_SOURCES): Temporarly
comment out folder.c, mbox.c and url.c.
Remove pop3_debug.c.
Add pop3_trace.c.
* libproto/pop/pop3_capa.c: Rewrite.
* libproto/pop/pop3_create.c: Rewrite.
* libproto/pop/pop3_iterator.c: Rewrite.
* libproto/pop/pop3_response.c: Rewrite.
* libproto/pop/pop3_sendline.c: Rewrite.
* libproto/pop/pop3_stls.c: Rewrite.
* libproto/pop/pop3_stream.c: Rewrite.
* libproto/pop/pop3_apop.c: Reflect changes to the pop3 framework.
* libproto/pop/pop3_carrier.c: Likewise.
* libproto/pop/pop3_connect.c: Likewise.
* libproto/pop/pop3_dele.c: Likewise.
* libproto/pop/pop3_destroy.c: Likewise.
* libproto/pop/pop3_disconnect.c: Likewise.
* libproto/pop/pop3_list.c: Likewise.
* libproto/pop/pop3_lista.c: Likewise.
* libproto/pop/pop3_noop.c: Likewise.
* libproto/pop/pop3_pass.c: Likewise.
* libproto/pop/pop3_quit.c: Likewise.
* libproto/pop/pop3_retr.c: Likewise.
* libproto/pop/pop3_readline.c: Likewise.
* libproto/pop/pop3_rset.c: Likewise.
* libproto/pop/pop3_stat.c: Likewise.
* libproto/pop/pop3_top.c: Likewise.
* libproto/pop/pop3_uidl.c: Likewise.
* libproto/pop/pop3_uidla.c: Likewise.
* libproto/pop/pop3_user.c: Likewise.

* examples/pop3client.c: Implement the stls comand.
(main) [WITH_TLS]: Call mu_init_tls_libs.
(com_verbose): Redo verbose support.
(com_capa): Implement "reread" option.

Sergey Poznyakoff authored 2010-09-08 16:11:36 +0300

Move libproto/include/(*)0.h to include/mailutils/sys/\1.h
Remove libproto/include.

Sergey Poznyakoff authored 2010-09-08 16:11:36 +0300

* pop3d/pop3d.h (POP_MAXCMDLEN, BUFFERSIZE): Remove constants.
(_QNX_SOURCE): Remove define.

Sergey Poznyakoff authored 2010-09-08 16:11:36 +0300

* pop3d/apop.c (pop3d_apopuser, pop3d_apopuser): Remove statically
allocated buffers.
* pop3d/pop3d.c (pop3d_mainloop): Likewise.
* pop3d/user.c (pop3d_begin_session): Likewise.

Sergey Poznyakoff authored 2010-09-08 16:11:36 +0300

* pop3d/apop.c (pop3d_apopuser): Fix highly ineffective resource
usage in both branches.
* pop3d/pop3d.c: Remove misleading comment.

Sergey Poznyakoff authored 2010-09-08 16:11:35 +0300

* libmu_auth/lbuf.c: Removed.
* libmu_auth/lbuf.h: Removed.
* libmu_auth/Makefile.am : Remove lbuf stuff.

* include/mailutils/sys/gsasl-stream.h: New file.
* include/mailutils/sys/Makefile.am: Add gsasl-stream.h.
* include/mailutils/gsasl.h (mu_gsasl_stream_create): Remove.
(gsasl_encoder_stream, gsasl_decoder_stream): New prototypes.

* libmu_auth/gsasl.c: Rewrite.

* imap4d/authenticate.c (auth_data): Remove.
Use struct imap4d_auth instead.
(_auth_try): Use new authentication API.
(imap4d_authenticate): Likewise.
* imap4d/imap4d.h (util_register_event, util_event_remove)
(util_run_events): Remove.
(imap4d_auth_handler_fp): Change prototype.
(imap4d_auth): New struct.
(imap4d_auth_result): New enum.
* imap4d/io.c (io_format_completion_response)
(io_stream_completion_response): New functions.
(io_completion_response): Rewrite using io_format_completion_response.
* imap4d/util.c (sc2string): Remove leftover prototype.
(util_register_event, util_event_remove)
(util_run_events): Remove.
* imap4d/auth_gsasl.c: Revamp using new streams and the new
authentication interface.
* imap4d/auth_gss.c: Likewise (though yet untested).

* mailbox/xscript-stream.c (_xscript_ctl): Remove unused variables.

Sergey Poznyakoff authored 2010-09-08 16:11:35 +0300

* mailbox/fltstream.c (filter_wr_flush): Fix erroneous conditional.
(filter_wait): New method.
(mu_filter_stream_create): Set wait method.

* imap4d/io.c: New source.
* imap4d/Makefile.am: Add io.c
* imap4d/*: Update I/O function calls.

Sergey Poznyakoff authored 2010-09-08 16:11:35 +0300

* pop3d/extra.c (pop3d_setio): Use the canonic name of the CRLF
filter, instead of the legacy "rfc822".
* imap4d/fetch.c: Likewise.
* imap4d/util.c: Likewise.

Sergey Poznyakoff authored 2010-09-08 16:10:49 +0300

* include/mailutils/stream.h (MU_IOCTL_SWAP_STREAM): New ioctl op.
(mu_stream_seterr): New function.
* include/mailutils/sys/stream.h (_MU_SWAP_FIRST_ONLY)
(_MU_SWAP_IOCTL_MUST_SUCCEED): New defines.
(_mu_stream_swap_streams): New proto.
* mailbox/amd.c (amd_body_stream_read): Always update *pnread.
* mailbox/errors (MU_ERR_NO_TRANSPORT): New error code.

* mailbox/stream.c (_stream_seterror): Rename to mu_stream_seterr,
remove static qualifier. All uses updated.
(_mu_stream_swap_streams): New function.

* mailbox/filter_iconv.c (_icvt_ioctl): Implement MU_IOCTL_SWAP_STREAM.
* mailbox/iostream.c (_iostream_ctl): Likewise.
* mailbox/xscript-stream.c (_xscript_ctl): Likewise.

* pop3d/extra.c (real_istream, real_ostream): Remove statics.
(pop3d_init_tls_server): Use MU_IOCTL_SWAP_STREAM to replace
bottom-level transport layers without recreating the entire stream
chain.

Sergey Poznyakoff authored 2010-09-08 16:10:48 +0300

If a stream takes another stream(s) as a transport, it
always increases its reference count.  This means that
when that stream is destroyed the underlying stream is
not destroyed by default (its refcount is decreased, that's
all). To force destruction of the underlying stream,
the caller must explicitly unreference it after creating
the stream that uses it (or give its creation function
the MU_STREAM_AUTOCLOSE flag, if it supports flags).

Similarly, if a stream uses a non-stream object (e.g. a file
descriptor) as the transport, it will not destroy it, unless
it has been created with the MU_STREAM_AUTOCLOSE flag. This
differs from the previous behavior.

The MU_STREAM_NO_CHECK and MU_STREAM_NO_CLOSE flags are removed.

* examples/base64.c (main): Call mu_filter_create with the
MU_STREAM_AUTOCLOSE flag.
* examples/mimetest.c (message_display_parts): Likewise.
* examples/murun.c (main): Unref the input stream after
passing it to mu_filter_prog_stream_create.
* imap4d/fetch.c (fetch_io): Update arguments to mu_filter_create
and mu_memory_stream_create.
* imap4d/preauth.c (decode64_buf)
(do_preauth_ident): Update arguments to mu_memory_stream_create and
mu_tcp_stream_create.
* imap4d/util.c (util_setio): Update arguments to mu_filter_create and
mu_stdio_stream_create.
* include/mailutils/stream.h (MU_STREAM_NO_CLOSE)
(MU_STREAM_NO_CHECK): Remove.
(MU_STREAM_AUTOCLOSE): New flag.
(mu_iostream_create): Remove the `flags' argument.
* libmu_argp/muinit.c (mu_app_init): Update arguments to
mu_stdio_stream_create.
* libmu_auth/ldap.c: Update arguments to
mu_memory_stream_create and mu_filter_create.
* libmu_auth/tls.c (_tls_io_close, _tls_close): Always try to close the
transport stream. Rely on refcount to protect it.
(_tls_io_done, _tls_done): Likewise, always unreference it.
(_mu_tls_io_stream_create): Increase reference counter on the
transport stream, unless MU_STREAM_AUTOCLOSE is requested.
(_mu_tls_stream_create): Rewrite using the new MU_STREAM_AUTOCLOSE
logic.
* libmu_sieve/extensions/spamd.c (spamd_connect_tcp): Update arguments to
mu_tcp_stream_create.
(spamd_connect_tcp): Update arguments to mu_socket_stream_create and
mu_filter_create.
* libmu_sieve/extensions/vacation.c (build_mime): Update arguments to
mu_filter_create.
* mail/decode.c (display_submessage): Update arguments to
mu_filter_create.
* mailbox/attachment.c (mu_message_save_attachment): Update arguments to
mu_filter_create.
* mailbox/cfg_driver.c (mu_cfg_tree_reduce): Update arguments to
mu_stdio_stream_create.
* mailbox/dbgstream.c (_dbg_done): Use MU_STREAM_AUTOCLOSE bit to
decide whether to destroy the debug object.
(mu_dbgstream_create): Use MU_STREAM_AUTOCLOSE instead of
MU_STREAM_NO_CLOSE.
* mailbox/file_stream.c (fd_close): Use MU_STREAM_AUTOCLOSE bit to
decide whether to close the descriptor.
(mu_file_stream_create): Force MU_STREAM_AUTOCLOSE bit.
* mailbox/filter.c (filter_create_rd, filter_create_wr): Change
substream creation logic.
* mailbox/filter_iconv.c (_icvt_close): Always try to close the
transport stream. Rely on refcount to protect it.
(_icvt_done): Ditto for destroying it.
(mu_filter_iconv_create): Increase refcount on the transport stream,
unless MU_STREAM_AUTOCLOSE is requested.
* mailbox/fltstream.c (filter_done): Always dereference the
transport stream. Rely on refcount to protect it.
(filter_close): Ditto for closing it.
(mu_filter_stream_create): Increase refcount on the transport stream,
unless MU_STREAM_AUTOCLOSE is requested.
* mailbox/iostream.c (_iostream_close) : Always try to close the
transport stream. Rely on refcount to protect it.
(_iostream_done): Ditto for closing it.
(mu_iostream_create): Remove the use of MU_STREAM_NO_CLOSE.
* mailbox/mimehdr.c (mu_mimehdr_decode_param): Remove the use of
MU_STREAM_NO_CLOSE.
* mailbox/mutil.c (mu_decode_filter) Change substream creation logic.
* mailbox/prog_stream.c (_prog_open): Use MU_STREAM_AUTOCLOSE bit
in arguments to mu_stdio_stream_create.
(mu_filter_prog_stream_create): Increase refcount on the transport
(input) stream.
* mailbox/rfc2047.c (mu_rfc2047_decode): Dereference in_stream after
passing it to mu_decode_filter.
Pass MU_STREAM_AUTOCLOSE in flags to mu_filter_create.
* mailbox/socket_stream.c (mu_socket_stream_create): Force
MU_STREAM_AUTOCLOSE bit.
* mailbox/streamref.c (_streamref_close): Always close the
transport stream (refcount will protect it, if necessary).
(mu_streamref_create_abridged): Mask out the MU_STREAM_AUTOCLOSE bit.
* mailbox/temp_file_stream.c (mu_temp_file_stream_create: Force
MU_STREAM_AUTOCLOSE bit.
* mailbox/xscript-stream.c (_xscript_close): Always close the
transport stream (refcount will protect it, if necessary).
(mu_xscript_stream_create): Increase refcounts on both underlying streams,
unless MU_STREAM_AUTOCLOSE is set.
* pop3d/extra.c (pop3d_setio): Remove uses of MU_STREAM_NO_CLOSE.

* examples/nntpclient.c: Remove uses of MU_STREAM_NO_CHECK/MU_STREAM_NO_CLOSE.
* examples/pop3client.c: Likewise.
* libmu_auth/gsasl.c: Likewise.
* libproto/nntp/nntp_stream.c: Likewise.
* libproto/pop/pop3_stream.c: Likewise.
* mailbox/tcp.c: Likewise.
* mailbox/vartab.c: Likewise.
* mh/mh_list.c: Likewise.
* mimeview/mimeview.c: Likewise.
* mh/mhn.c: Likewise.
(edit_mime): Use MU_STREAM_AUTOCLOSE.

Bugfixes:

* mailbox/fltstream.c (init_iobuf): Fix input initialization.
(filter_write_internal): Bugfix.
* mailbox/stream.c (_stream_buffer_freespace): New macro.
(_stream_buffer_is_full): New macro.
(BUFFER_FULL_P): Remove, use _stream_buffer_is_full instead. All callers
updated.
(_stream_flush_buffer): Operation for full buffered streams does not
depend on the `all' flag.
(mu_stream_write): Fix calculation of the bytes available in the
buffer.

Sergey Poznyakoff authored 2010-09-08 16:10:48 +0300

* include/mailutils/stream.h (MU_STREAM_RDTHRU)
(MU_STREAM_WRTHRU,MU_IOCTL_SET_TRANSPORT): New flags.
(mu_xscript_stream_create, mu_iostream_create)
(mu_dbgstream_create): New prototypes.
* include/mailutils/sys/dbgstream.h: New header.
* include/mailutils/sys/iostream.h: New header.
* include/mailutils/sys/xscript-stream.h: New header.
* include/mailutils/sys/Makefile.am (sysinclude_HEADERS): Add
dbgstream.h, iostream.h and xscript-stream.h
* mailbox/dbgstream.c: New file.
* mailbox/iostream.c: New file.
* mailbox/xscript-stream.c: New file.
* mailbox/Makefile.am (libmailutils_la_SOURCES): Add dbgstream.c,
iostream.c and xscript-stream.c
* mailbox/filter_iconv.c (_icvt_ioctl): Simplify the declaration
of ptrans.
* mailbox/mapfile_stream.c (_mapfile_ioctl): Likewise.
* mailbox/memory_stream.c (_memory_ioctl): Likewise.
* mailbox/prog_stream.c (_prog_ioctl): Likewise.
* mailbox/tcp.c (_tcp_ioctl): Likewise.
* mailbox/fltstream.c (filter_ctl): Likewise.
(filter_read_through, filter_write_through): New methods.
(mu_filter_stream_create): Allow for use of MU_STREAM_RDTHRU
and MU_STREAM_WRTHRU to create two-way filters (writing
triggers filtering while reading is transparent or vice versa).
* pop3d/extra.c (istream, ostream): Remove globals.
(iostream): New variable.
(real_istream, real_ostream): New variables.
(pop3d_setio): Set transcript stream on top of the I/O one,
if required.
(pop3d_init_tls_server): Rewrite. Revert the meaning of the
return code to match the usual convention (0 - success).
(transcript): Removed.
(pop3d_outf): Remove calls to transcript.
* pop3d/pop3d.h (istream, ostream): Remove externs.
(iostream): New extern.
* pop3d/retr.c: Use iostream, instear of ostream.
* pop3d/top.c: Likewise.
* pop3d/stls.c: Update the call to pop3d_init_tls_server.

* mailbox/stream_vprintf.c (mu_stream_vprintf): Fix return
value to match the usual convention.

Sergey Poznyakoff authored 2010-09-08 16:10:48 +0300

STLS in pop3d is already working.

* include/mailutils/sys/tls-stream.h: New header.
* include/mailutils/tls.h (mu_tls_stream_create)
(mu_tls_stream_create_client)
(mu_tls_stream_create_client_from_tcp): Remove.
(mu_tls_server_stream_create, mu_tls_client_stream_create): New
protos.
* libmu_auth/tls.c: Rewrite.

* imap4d/util.c: Use mu_tls_server_stream_create.
* libproto/pop/mbox.c: Use mu_tls_server_stream_create/
mu_tls_client_stream_create.
* libproto/pop/pop3_stls.c: Use mu_tls_client_stream_create.
* libproto/imap/folder.c: Use mu_tls_client_stream_create.

* pop3d/capa.c (pop3d_capa): CAPA is allowed in both states.
* pop3d/extra.c (pop3d_setio): Rewrite.

Sergey Poznyakoff authored 2010-09-08 16:10:48 +0300

* mailbox/amd.c (amd_body_stream_readdelim): Fix the logic.
* mailbox/message.c (_message_stream_seek): Fix seeks in
backward direction.
* pop3d/top.c: Fix output of the body.

Sergey Poznyakoff authored 2010-09-08 16:10:48 +0300

* imap4d/append.c: Add a comment.
* imap4d/imap4d.h (imap4d_child_signal_setup): New proto.
* libmu_sieve/extensions/pipe.c (sieve_action_pipe): Use
mu_stream_copy, rewrite error handling.
* libmu_sieve/extensions/spamd.c (spamd_send_message): se
mu_stream_copy.
(spamd_test): Honor dry-run mode.
* libmu_sieve/extensions/vacation.c (build_mime): Remove
misleading FIXME. Use mu_stream_copy.
* libmu_sieve/extensions/moderator.c (moderator_message_get_part): Use
mu_body_get_streamref.
* examples/header.c (hstream): New global.
(cmd_load, cmd_free, cmd_remove, cmd_insert): Discard hstream.
(cmd_readline): Use hstream. Obtain it using mu_stream_readline.
* libproto/mbox/mbox.c (mbox_envelope_date): Remove trailing
newline from the obtained envelope line.
* mailbox/amd.c (amd_body_stream_seek): Allow for off == size
(imprtant for empty bodies).

Sergey Poznyakoff authored 2010-09-08 16:10:48 +0300

* libmu_sieve/actions.c (mime_create_quote): Use mu_stream_copy.

Sergey Poznyakoff authored 2010-09-08 16:10:47 +0300

* pop3d/pop3d.h: Include filter.h.
(istream, ostream): New externs.
* pop3d/extra.c (pop3d_setio): Set a crlf filter on the
output stream. This allows to forget about \r in the rest of
the code.
* pop3d/retr.c (pop3d_retr): Use mu_stream_copy instead of
copying streams manually.
* pop3d/top.c (pop3d_top): Rewrite using streamrefs and
mu_stream_copy.
* all sources: Use \n instead of \r\n in output strings.

Sergey Poznyakoff authored 2010-09-08 16:10:47 +0300

* examples/mta.c (finalize_option): New variable.
(message_finalize): Don't modify the message if finalize_option
is 0.
* include/mailutils/sys/message.h (_mu_message)
<orig_header_size>: New member.
* include/mailutils/sys/mime.h (_mu_mime) <part_stream>: New member.
* libmu_sieve/actions.c (mime_create_reason)
(mime_create_ds): Use mu_body_get_streamref.
(mime_create_quote): Use mu_body_get_streamref/mu_message_get_streamref.
(sieve_action_reject): Set the To: header.
* mailbox/message.c (mu_message_get_header): Set orig_header_size.
(mu_message_get_body): Use orig_header_size instead of relying
on mu_header_size.
* mailbox/mime.c (_mime_part_size): New static.
(_mime_body_stream_size): New method.
(_mime_body_seek): Rename to _mime_body_stream_seek.
(_mime_body_read): Rewrite and rename to _mime_body_stream_read.
(_mime_body_ioctl): Rename to _mime_body_stream_ioctl.
(create_mime_body_stream): Take mu_mime_t as the 2nd parameter.
Initialize sp->stream.size and sp->mime.
(_mime_body_size): Rewrite using _mime_part_size.
(mu_mime_destroy): Destroy part_stream.
* sieve/testsuite/Redirect: Reflect new mta behavior.
* sieve/testsuite/Reject: Likewise.

Sergey Poznyakoff authored 2010-09-08 16:10:47 +0300

* libproto/mbox/mbox.c (_msg_stream_setup): Fix abridgement limits.
(mbox_expunge_unlocked): Don't write extra newline after each
message.
* mailbox/stream.c (mu_stream_seek): ESPIPE does not constitute
a permanent error.

Sergey Poznyakoff authored 2010-09-08 16:09:55 +0300

* mailbox/mime.c (mu_mime_get_part): Fix the end limit for the
part's abridgement.
* mailbox/streamref.c (_streamref_seek): Fix ESPIPE condition.

Sergey Poznyakoff authored 2010-09-08 16:09:55 +0300

* include/mailutils/message.h (mu_message_set_stream): Mark as deprecated.
* pop3d/retr.c (pop3d_retr): Rewind the stream obtained from
mu_message_get_stream.
* pop3d/top.c (pop3d_top): Rewind the streams obtained from
mu_body_get_stream and mu_header_get_stream.

Sergey Poznyakoff authored 2010-09-08 16:09:55 +0300

* mailbox/base64.c (_base64_encoder): Fix the use
of uninitialized variable.
* mailbox/rfc2047.c (mu_rfc2047_encode): Rewind the
input stream before reading from it.

Sergey Poznyakoff authored 2010-09-08 16:09:55 +0300

* mailbox/filter_iconv.c (mu_filter_iconv_create): Set filter flags.
* mailbox/fltstream.c (mu_filter_stream_create): Likewise.
* mailbox/qpflt.c (_qp_decoder): Handle '_'.

Sergey Poznyakoff authored 2010-09-08 16:09:55 +0300

* examples/base64.c (c_copy): Handle printable mode (-p).
(reset_line_length): New function.
(main): New option -lN sets the maximum length for output
lines to N (0 means unlimited).
* mailbox/testsuite/mailbox/base64.exp: Use -l0 in the
Decode test.

Sergey Poznyakoff authored 2010-09-08 16:09:55 +0300

* include/mailutils/header.h (mu_header_invalidate): New proto.
* mailbox/header.c (mu_header_invalidate): New function.

* libmu_argp/muinit.c (get_canonical_name): Avoid coredumping
if argp_program_version is NULL.
* mailbox/base64.c (mu_base64_decode): Fix inconsistent return
code.
* mailbox/debug.c (mu_debug_destroy): Allow for debug->stream == NULL.

* mailbox/mapfile_stream.c (_mapfile_truncate): Incorrect size
was used when unmapping.
* mailbox/message.c (mu_message_create_copy): Use a temporary
memory stream.
(mu_message_get_body): Comment out the check for MESSAGE_INTERNAL_STREAM.
(_message_get_stream): Initialize message header and body.

* mailbox/progmailer.c (mu_progmailer_send): Check  return
from the mu_header_get_streamref.

* mailbox/stream.c (_stream_scandelim, _stream_readdelim): If
size is 0, return MU_ERR_BUFSPACE.

Sergey Poznyakoff authored 2010-09-08 16:09:54 +0300

Readmsg passes all tests successfully.

* readmsg/msglist.c (msglist): Check return from mu_message_get_streamref.
* readmsg/readmsg.c (print_header, print_body): Check return values.

Sergey Poznyakoff authored 2010-09-08 16:09:54 +0300

* mailbox/header.c (header_seek): Do not dereference
hstr->hdr->size directly, because it may not be initialized yet.
Use mu_header_size instead.
* mailbox/message.c (_message_stream_readdelim): Break the loop
if mu_stream_readdelim returned error or EOF.

Sergey Poznyakoff authored 2010-09-08 16:09:54 +0300

Imap4d successfully passes all tests.

* imap4d/close.c (imap4d_close0): Fix improper use of MU_STREAM_READ
(flags changed their semantics since then).
* imap4d/fetch.c: Send \n terminated lines, rely on filters to recode
line terminators to \r\n.
(fetch_io): Rewrite using CRLF encoder and util_copy_out.
(_frt_header_fields): Rewind memory stream before reading from it.

* imap4d/auth_gsasl.c: Send \n terminated lines, rely on filters
to recode line terminators to \r\n.
* imap4d/auth_gss.c: Likewise.
* imap4d/capability.c: Likewise.
* imap4d/copy.c: Likewise.
* imap4d/id.c: Likewise.
* imap4d/idle.c: Likewise.
* imap4d/list.c: Likewise.
* imap4d/namespace.c: Likewise.
* imap4d/preauth.c: Likewise.
* imap4d/search.c: Likewise.
* imap4d/status.c: Likewise.
* imap4d/store.c: Likewise.
* imap4d/select.c: Likewise.
(imap4d_select_status): Fix improper use of MU_STREAM_READ.

* imap4d/util.c: Send \n terminated lines, rely on filters
to recode line terminators to \r\n.
(util_setio): Apply CRLF filters to both input and output streams
(in opposite directions).
(util_copy_out): New function.
(remove_cr): Remove.
* imap4d/imap4d.h (util_copy_out): New prototype.

Sergey Poznyakoff authored 2010-09-08 16:09:54 +0300

* libproto/mbox/mbox.c (append_message_to_stream): Add \n after each
message (scanner relies on this to determine message sizes).

Sergey Poznyakoff authored 2010-09-08 16:09:54 +0300